How to call Python from a boost thread?

Question

I have a Python app that calls a C++ boost python library and it all works. However, I have a callback C++ to Python scenario where C++ from a boost thread calls python and I get an access violation on the C++ side. If I do exactly the same callback using the python thread it works perfectly. Therefore I suspect that I can not simply callback Python from C++ using a boost thread but need to do something extra for it to work?

Answer 1

The most likely culprit is that the Global Interpreter Lock (GIL) is not being held by a thread when it is invoking Python code, resulting in undefined behavior. Verify all paths that make direct or indirect Python calls, acquire the GIL before invoking Python code.

---

The GIL is a mutex around the CPython interpreter. This mutex prevents parallel operations to be performed on Python objects. Thus, at any point in time, a max of one thread, the one that has acquired the GIL, is allowed to perform operations on Python objects. When multiple threads are present, invoking Python code whilst not holding the GIL results in undefined behavior.

C or C++ threads are sometimes referred to as alien threads in the Python documentation. The Python interpreter has no ability to control the alien thread. Therefore, alien threads are responsible for managing the GIL to permit concurrent or parallel execution with Python threads. One must meticulously consider:

• The stack unwinding, as Boost.Python may throw an exception.
• Indirect calls to Python, such as copy-constructors or destructors

One solution is to wrap Python callbacks with a custom type that is aware of GIL management.

---

Using a RAII-style class to manage the GIL provides an elegant exception-safe solution. For example, with the following with_gil class, when a with_gil object is created, the calling thread acquires the GIL. When the with_gil object is destructed, it restores the GIL state.

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

And its usage:

{
  with_gil gil;                      // Acquire GIL.
  // perform Python calls, may throw
}                                    // Restore GIL.

With being able to manage the GIL via with_gil, the next step is to create a functor that properly manages the GIL. The following py_callable class will wrap a boost::python::object and acquire the GIL for all paths in which Python code is invoked:

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

By managing the boost::python::object on the free-space, one can freely copy the shared_ptr without having to hold the GIL. This allows for us to safely use the default generated copy-constructor, assignment operator, destructor, etc.

One would use the py_callable as follows:

// thread 1
boost::python::object object = ...; // GIL must be held.
py_callable callback(object);       // GIL no longer required.
work_queue.post(callback);

// thread 2
auto callback = work_queue.pop();   // GIL not required.
// Invoke the callback.  If callback is `py_callable`, then it will
// acquire the GIL, invoke the wrapped `object`, then release the GIL.
callback(...);   

---

Here is a complete example demonstrating having a Python extension invoke a Python object as a callback from a C++ thread:

#include <memory>  // std::shared_ptr
#include <thread>  // std::this_thread, std::thread
#include <utility> // std::forward
#include <boost/python.hpp>

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::def("call_later",
    +[](int delay, python::object object) {
      // Create a thread that will invoke the callback.
      std::thread thread(+[](int delay, py_callable callback) {
        std::this_thread::sleep_for(std::chrono::seconds(delay));
        callback("spam");
      }, delay, py_callable{object});
      // Detach from the thread, allowing caller to return.
      thread.detach();
  });
}

Interactive usage:

>>> import time
>>> import example
>>> def shout(message):
...     print message.upper()
...
>>> example.call_later(1, shout)
>>> print "sleeping"; time.sleep(3); print "done sleeping"
sleeping
SPAM
done sleeping