Boost.Python create new reference to existing Python object from C++

Question

I am wrapping a C++ class X using Boost.Python. At the moment an object of this class is created, I would like to insert an additional reference to this object into the local namespace (so that I can refer to his newly created object by a fixed name, let's say lastX). I tried to do this in the X::X() constructor using

X::X() 
   {
   boost::python::object locals(boost::python::borrowed(PyEval_GetLocals()));
   boost::python::object me(this);
   locals["lastX"]=me;
   }

but this does not work (lastX gets created but it refers to nothing; printing it from Python leads to a segfault). I should probably use my own init function but I do not know how to get a reference to the newly created Python object there either.

Any ideas? Thanks.

Answer 1

To accomplish this, one must modify another frame on the call stack. Be warned, this is dependent on the Python implementation. For example, in Python 2.7, the inspect module and sys.settrace() can be used to modify locals() on a specific frame.

I would highly recommend using a Python solution, as was done in this answer, and monkey patch the desired class' __init__ function. For instance, the following would patch the Spam class to insert a variable named last_spam that references the newly constructed Spam instance into the caller's frame:

def _patch_init(cls, name):
    cls_init = getattr(cls, '__init__', None)
    def patch(self):
        caller_frame = inspect.currentframe(1)
        new_locals = caller_frame.f_locals
        new_locals[name] = self
        _force_locals(caller_frame, new_locals)
        if cls_init:
            cls_init(self)
    setattr(cls, '__init__', patch)

_patch_init(Spam, 'last_spam')

spam = Spam()
assert(spam is last_spam)

---

Nevertheless, the same can be accomplished with Boost.Python. To accomplish this, one must:

• Modify a frame's locals().
• Access the self instance during object construction.

Be warned, these can be fairly advance topics.

Modify a frame's locals()

This is the same approach used in this answer, which is dependent on the Python implementation, but written in C++. In most execution paths, a frame's locals() cannot have new variables written into it. However, when system tracing is enabled, frame tracing functions, often used by debuggers and other tools, can modify a frame's locals().

/// @brief Trace signature type.  Boost.Python requires MPL signature for
///        custom functors.
typedef boost::mpl::vector<
  boost::python::object, // return
  boost::python::object, // frame
  boost::python::object, // event
  boost::python::object // argt
> trace_signature_type;

/// @brief A noop function for Python.  Returns None.
boost::python::object noop(
  boost::python::tuple /* args */,
  boost::python::dict /* kw */
)
{
  return boost::python::object();
}

/// @brief Inject new_locals into the provided frame.
void inject_locals_into_frame(
  boost::python::object frame,
  boost::python::dict new_locals
)
{
  namespace python = boost::python;
  // Force tracing by setting the global tracing function to any non-None
  // function.
  // # if not sys.gettrace():
  if (!PyThreadState_Get()->c_tracefunc)
  {
    // Use the sys.settrace function to prevent needing to re-implement the
    // trace trampoline.
    //   # import sys
    python::object sys(python::handle<>(PyImport_ImportModule("sys")));
    //   # sys.settrace(lambda *args, **keys: None)
    sys.attr("__dict__")["settrace"](python::raw_function(&detail::noop));
  }

  // Create trace function.
  //   # def trace(frame, event, arg):
  python::object trace = python::make_function([new_locals](
        python::object frame,
        python::object /* event */,
        python::object /* arg */
      )
      {
        // Update the frame's locals.
        //   # frame.f_locals.update(new_locals)
        frame.attr("f_locals").attr("update")(new_locals);

        // Set the frame to use default trace, preventing this
        // trace functor from resetting the locals on each trace call.
        //   # del frame.f_trace
        frame.attr("f_trace").del();

        //   # return None
        return boost::python::object();
      },
    python::default_call_policies(),
    trace_signature_type());

  // Set the frame to use the custom trace.
  //   # frame.f_trace = trace
  frame.attr("f_trace") = trace;
}

With the above code, one can use inject_into_frame_locals() to update a given frame's locals. For example, the following will add a variable x that references 42 into the current frame:

// Create dictionary that will be injected into the current frame.
namespace python = boost::python;
python::dict new_locals;
new_locals["x"] = 42;

// Get a handle to the current frame.
python::object frame(python::borrowed(
  reinterpret_cast<PyObject*>(PyEval_GetFrame())));

// Set locals to be injected into the frame.
inject_into_frame_locals(frame, new_locals);

---

Access the self instance during object construction.

Given a C++ object, one cannot use the Boost.Python API to locate the Python object in which the object is held. Thus, one must access self as Boost.Python is constructing the object.

There are a few customization points:

• Modify the class being exposed to accept PyObject* during construction. One can have Boost.Python provide the PyObject* instance be specializing has_back_reference:

  struct foo
  {
    // Constructor.
    foo(PyObject* self)
    {
      namespace python = boost::python;
      python::handle<> handle(python::borrowed(self));
      trace::inject_into_current_frame("last_foo", python::object(handle));
    }
  
    // Boost.Python copy constructor.
    foo(PyObject* self, const foo& /* rhs */)
    {
      namespace python = boost::python;
      python::handle<> handle(python::borrowed(self));
      trace::inject_into_current_frame("last_foo", python::object(handle));
    }
  };
  
  namespace boost {
  namespace python {
    // Have Boost.Python pass PyObject self to foo during construction.
    template <>
    struct has_back_reference<foo>
      : boost::mpl::true_
    {};
  } // namespace python
  } // namespace boost
  
  BOOST_PYTHON_MODULE(example)
  {
    namespace python = boost::python;
    python::class_<foo>("Foo", python::init<>());
  }
  

• Expose the T as being held by a HeldType that derives from T. When the HeldType is publicly derived from the T, it will be provided PyObject* during construction:

  struct bar {};
  
  struct bar_holder
    : public bar
  {
    bar_holder(PyObject* self)
    {
     namespace python = boost::python;
      python::handle<> handle(python::borrowed(self));
      trace::inject_into_current_frame("last_bar", python::object(handle));
    }
  
    bar_holder(PyObject* self, const bar& /* rhs */)
    {
      namespace python = boost::python;
      python::handle<> handle(python::borrowed(self));
      trace::inject_into_current_frame("last_bar", python::object(handle));
   }
  };
  
  BOOST_PYTHON_MODULE(example)
  {
    namespace python = boost::python;
    python::class_<bar, bar_holder>("Bar", python::init<>());
  }
  

• Suppress Boost.Python from generating a default initializer with boost::python::no_init, then register a custom factory function as the __init__ method with custom policies. One key function for customizing object construction is the boost::python::make_constructor function. When provided a pointer to a C++ function or pointer-to-member-function, it will return a Python callable object, which when called, will invoke the function and create the Python object. However, Boost.Python attempts to hide Python-specific details from the C++ functions, and thus self is not explicitly provided. Nevertheless, one can use a a custom CallPolicy, and access the Python arguments within either the precall or postcall functions. In order to access the self argument, one has to be aware of an implementation detail with make_constructor's policies, which offsets argument access by 1. For instance, when attempting to access the self argument that resides at index 0, one must request the -1 index.

  // Mockup models.
  class spam {};
  
  // Factor functions for the models will be necessary, as custom constructor
  // functions will be needed to provide a customization hook for our models.
  spam* make_spam() { return new spam(); }
  
  template <typename BasePolicy = boost::python::default_call_policies>
  struct custom_policy
    : BasePolicy
  {     
    template <typename ArgumentPackage>
    PyObject* postcall(const ArgumentPackage& args, PyObject* result)
    {
      namespace python = boost::python;
  
      // Chain to base policy.
      result = BasePolicy::postcall(args, result);
  
      // self is the first argument.  It is an implementation detail that
      // the make_constructor policy will offset access by 1.  Thus, to 
      // access the actual object at index 0 (self), one must use -1.
      python::object self(python::borrowed(
        get(boost::mpl::int_<-1>(), args)));
  
      return result;
    }
  };
  
  BOOST_PYTHON_MODULE(example)
  {
    namespace python = boost::python;
    python::class_<spam>("Spam", python::no_init)
      .def("__init__", python::make_constructor(
        &make_spam, 
        custom_policy<>()))
     ;
  }
  

• Once again, the default initializer can be suppressed and the object returned by make_constructor can be decorated. When the __init__ method is invoked, the decorator object will be invoked where it will be provided all the arguments for the initializer, including self. The decorator will need to delegate to the object returned from make_constructor. Use boost::python::make_function() to transform a C++ functor to a callable Python object. Please note that the make_function documentation does not state it supports custom functors. However, it is used internally for functor support.

  // Mockup models.
  class egg {};
  
  // Factor functions for the models will be necessary, as custom constructor
  // functions will be needed to provide a customization hook for our models.
  egg* make_egg() { return new egg(); }
  
  template <typename Fn>
  class custom_constructor
  {
  public:
  
    typedef boost::python::object result_type;
  
  public:
  
    custom_constructor(Fn fn)
      : constructor_(boost::python::make_constructor(fn))
    {}
  
    /// @brief Initialize python object.
    template <typename ...Args>
    result_type operator()(boost::python::object self, Args... args)
    {
      return constructor_(self, args...);
    }
  
  private:
    boost::python::object constructor_;
  };
  
  template <typename Fn>
  boost::python::object make_custom_constructor(Fn fn)
  {
    // Use MPL to decompose the factory function signature into the
    // desired Python object signature.
    typedef /* ... */ signature_type;
  
    // Create a callable python object from custom_constructor.
    return boost::python::make_function(
      custom_constructor<Fn>(fn),
      boost::python::default_call_policies(),
      signature_type());
  }
  
  BOOST_PYTHON_MODULE(example)
  {
    namespace python = boost::python;
    python::class_<egg>("Egg", python::no_init)
      .def("__init__", make_custom_constructor(&make_egg))
      ;
  }
  

---

Here is a complete example demonstrating all of the above approaches:

#include <boost/function_types/components.hpp>
#include <boost/mpl/insert_range.hpp>
#include <boost/python.hpp>
#include <boost/python/raw_function.hpp>

namespace trace {
namespace detail {

/// @brief Trace signature type.  Boost.Python requires MPL signature for
///        custom functors.
typedef boost::mpl::vector<
  boost::python::object, // return
  boost::python::object, // frame
  boost::python::object, // event
  boost::python::object  // arg
> trace_signature_type;

/// @brief A noop function for Python.  Returns None.
boost::python::object noop(
  boost::python::tuple /* args */,
  boost::python::dict /* kw */
)
{
  return boost::python::object();
}

} // namespace detail

/// @brief Inject new_locals into the provided frame.
void inject_into_frame_locals(
  boost::python::object frame,
  boost::python::dict new_locals
)
{
  namespace python = boost::python;
  // Force tracing by setting the global tracing function to any non-None
  // function.
  // # if not sys.gettrace():
  if (!PyThreadState_Get()->c_tracefunc)
  {
    // Use the sys.settrace function to prevent needing to re-implement the
    // trace trampoline.
    //   # import sys
    python::object sys(python::handle<>(PyImport_ImportModule("sys")));
    //   # sys.settrace(lambda *args, **keys: None)
    sys.attr("__dict__")["settrace"](python::raw_function(&detail::noop));
  }

  // Create trace function.
  //   # def trace(frame, event, arg):
  python::object trace = python::make_function([new_locals](
        python::object frame,
        python::object /* event */,
        python::object /* arg */
      )
      {
        // Update the frame's locals.
        //   # frame.f_locals.update(new_locals)
        frame.attr("f_locals").attr("update")(new_locals);

        // Set the frame to use default trace, preventing this
        // trace functor from resetting the locals on each trace call.
        //   # del frame.f_trace
        frame.attr("f_trace").del();

        //   # return None
        return boost::python::object();
      },
    python::default_call_policies(),
    detail::trace_signature_type());

  // Set the frame to use the custom trace.
  //   # frame.f_trace = trace
  frame.attr("f_trace") = trace;
}

/// @brief Helper function used to setup tracing to inject the key-value pair
///        into the current frame.
void inject_into_current_frame(
  std::string key,
  boost::python::object value)
{
  // If there is no key, just return early.
  if (key.empty()) return;

  // Create dictionary that will be injected into the current frame.
  namespace python = boost::python;
  python::dict new_locals;
  new_locals[key] = value;

  // Get a handle to the current frame.
  python::object frame(python::borrowed(
    reinterpret_cast<PyObject*>(PyEval_GetFrame())));

  // Set locals to be injected into the frame.
  inject_into_frame_locals(frame, new_locals);
}

} // namespace trace

/// APPROACH 1: has_back_reference

struct foo
{
  // Constructor.
  foo(PyObject* self)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(self));
    trace::inject_into_current_frame("last_foo", python::object(handle));
  }

  // Boost.Python copy constructor.
  foo(PyObject* self, const foo& /* rhs */)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(self));
    trace::inject_into_current_frame("last_foo", python::object(handle));
  }
};

namespace boost {
namespace python {
  // Have Boost.Python pass PyObject self to foo during construction.
  template <>
  struct has_back_reference<foo>
    : boost::mpl::true_
  {};
} // namespace python
} // namespace boost

/// APPROACH 2: custom holder

struct bar {};

struct bar_holder
  : public bar
{
  bar_holder(PyObject* self)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(self));
    trace::inject_into_current_frame("last_bar", python::object(handle));
  }

  bar_holder(PyObject* self, const bar& /* rhs */)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(self));
    trace::inject_into_current_frame("last_bar", python::object(handle));
  }
};

/// APPROACH 3: custom call policy

struct spam {};

/// @brief CallPolicy that injects a reference to the returned object
///        into the caller's frame.  Expected to only be used as a
//         policy for make_constructor.
template <typename BasePolicy = boost::python::default_call_policies>
struct inject_reference_into_callers_frame
  : BasePolicy
{
  inject_reference_into_callers_frame(const char* name)
    : name_(name)
  {}

  template <typename ArgumentPackage>
  PyObject* postcall(const ArgumentPackage& args, PyObject* result)
  {
    // Chain to base policy.
    result = BasePolicy::postcall(args, result);

    // self is the first argument.  It is an implementation detail that
    // the make_constructor policy will offset access by 1.  Thus, to 
    // access the actual object at index 0 (self), one must use -1.
    namespace python = boost::python;
    python::object self(python::borrowed(
      get(boost::mpl::int_<-1>(), args)));

    // Inject into the current frame. 
    trace::inject_into_current_frame(name_, self);

    return result;
  }

private:
  std::string name_;
};

// Factor functions for the models will be necessary, as custom constructor
// functions will be needed to provide a customization hook for our models.
spam* make_spam() { return new spam(); }

/// APPROACH 4: decorated constructor
//
struct egg {};

namespace detail {

/// @brief A constructor functor that injects the constructed object
///        into the caller's frame.
template <typename Fn>
class inject_constructor
{
public:

  typedef boost::python::object result_type;

public:

  /// @brief Constructor.
  inject_constructor(
    const char* name,
    Fn fn
  )
    : name_(name),
      constructor_(boost::python::make_constructor(fn))
  {}

  /// @brief Initialize the python objet.
  template <typename ...Args>
  result_type operator()(boost::python::object self, Args... args)
  {
    // Initialize the python object.
    boost::python::object result = constructor_(self, args...);

    // Inject a reference to self into the current frame.
    trace::inject_into_current_frame(name_, self);

    return result;
  }

private:
  std::string name_;
  boost::python::object constructor_;
};

} // namespace detail

/// @brief Makes a wrapper constructor (constructor that works with
///        classes inheriting from boost::python::wrapper).
template <typename Fn>
boost::python::object make_inject_constructor(
  const char* name,
  Fn fn)
{
  // Decompose the factory function signature, removing the return type.
  typedef typename boost::mpl::pop_front<
    typename boost::function_types::components<Fn>::type
  >::type components_type;

  // Python constructors take the instance/self argument as the first
  // argument, and returns None.  Thus, inject python::objects into the
  // signature type for both the return and 'self' argument.
  typedef typename boost::mpl::insert_range<
    components_type, 
    typename boost::mpl::begin<components_type>::type,
    boost::mpl::vector<boost::python::object, boost::python::object>
  >::type signature_type;

  // Create a callable python object from inject_constructor.
  return boost::python::make_function(
    detail::inject_constructor<Fn>(name, fn),
    boost::python::default_call_policies(),
    signature_type());
}

egg* make_egg() { return new egg(); }

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // APPROACH 1: has_back_reference
  python::class_<foo>("Foo", python::init<>());

  // APPROACH 2: custom holder
  python::class_<bar, bar_holder>("Bar", python::init<>());

  // APPROACH 3: custom call policy
  python::class_<spam>("Spam", python::no_init)
    .def("__init__", python::make_constructor(
      &make_spam,
      inject_reference_into_callers_frame<>("last_spam")))
    ;

  // APPROACH 4: decorated constructor
  python::class_<egg>("Egg", python::no_init)
    .def("__init__", make_inject_constructor("last_egg", &make_egg))
    ;
}

Interactive usage:

>>> import example
>>> foo = example.Foo()
>>> assert(foo is last_foo)
>>> bar = example.Bar()
>>> assert(bar is last_bar)
>>> spam = example.Spam()
>>> assert(spam is last_spam)
>>> egg = example.Egg()
>>> assert(egg is last_egg)