boost python: tie lifetime of argument to returned value using return_internal_reference

Question

I begin learning to use boost python and have a rookie question.

I would like to write a function that can tie the lifetime of its argument to its results, such that when I call r = func(a), the argument a will never be destroyed if I still have a reference of r. The documentation suggests using return_internal_reference call policy for this type of request. But does this require r to be an internal reference of a, as the name suggested?

In the (over-simplified) example below, suppose I want to tie the lifetime of input array a with the generated lambda function, which is not an internal reference of input a.

#include <functional>
#include <boost/python.hpp>
#include <boost/python/return_internal_reference.hpp>

using namespace std;
using namespace boost::python;

function<float(int)> func(const float* a) {
  return [=](int n) { return a[n]; };
}

BOOST_PYTHON_MODULE(test) {
  def("func", func, return_internal_reference<1>());
}

I hope to be able to do the following in python:

f = func(a)   # 'a' can be a temporary variable, say returned by another function
f(5)          # but 'a' should not be destroyed at this step, 
              # because its lifetime is tied to 'f'

When I tried to compile the code above, I got a wall of errors listed below, but if I remove the return_internal_reference<1>() call policy, the code compiles successfully.

I am pretty sure I used this call policy wrong, but am not sure how to make it right. Any pointer will be highly appreciated. Thanks a lot!

$ g++ -std=c++11 -shared Test.cc -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -L/opt/local/lib -lboost_python-mt -lpython2.7 -o test.so
In file included from /opt/local/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:52:0,
                 from /opt/local/include/boost/python/detail/invoke.hpp:63,
                 from /opt/local/include/boost/python/detail/caller.hpp:16,
                 from /opt/local/include/boost/python/object/function_handle.hpp:8,
                 from /opt/local/include/boost/python/converter/arg_to_python.hpp:19,
                 from /opt/local/include/boost/python/call.hpp:15,
                 from /opt/local/include/boost/python/object_core.hpp:14,
                 from /opt/local/include/boost/python/args.hpp:25,
                 from /opt/local/include/boost/python.hpp:11,
                 from Test.cc:2:
/opt/local/include/boost/python/detail/invoke.hpp: In instantiation of 'PyObject* boost::python::detail::invoke(boost::python::detail::invoke_tag_<false, false>, const RC&, F&, AC0&) [with RC = boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >; F = std::function<float(int)> (*)(const float*); AC0 = boost::python::arg_from_python<const float*>; PyObject = _object]':
/opt/local/include/boost/python/detail/caller.hpp:223:13:   required from 'PyObject* boost::python::detail::caller_arity<1u>::impl<F, Policies, Sig>::operator()(PyObject*, PyObject*) [with F = std::function<float(int)> (*)(const float*); Policies = boost::python::return_internal_reference<>; Sig = boost::mpl::vector2<std::function<float(int)>, const float*>; PyObject = _object]'
/opt/local/include/boost/python/object/py_function.hpp:38:33:   required from 'PyObject* boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*, PyObject*) [with Caller = boost::python::detail::caller<std::function<float(int)> (*)(const float*), boost::python::return_internal_reference<>, boost::mpl::vector2<std::function<float(int)>, const float*> >; PyObject = _object]'
Test.cc:14:1:   required from here
/opt/local/include/boost/python/detail/invoke.hpp:75:82: error: no match for call to '(const boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >) (std::function<float(int)>)'
     return rc(f( BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () BOOST_PP_INTERCEPT) ));
                                                                                  ^
In file included from /opt/local/include/boost/python/object/function_handle.hpp:8:0,
                 from /opt/local/include/boost/python/converter/arg_to_python.hpp:19,
                 from /opt/local/include/boost/python/call.hpp:15,
                 from /opt/local/include/boost/python/object_core.hpp:14,
                 from /opt/local/include/boost/python/args.hpp:25,
                 from /opt/local/include/boost/python.hpp:11,
                 from Test.cc:2:
/opt/local/include/boost/python/detail/caller.hpp: In instantiation of 'static const PyTypeObject* boost::python::detail::converter_target_type<ResultConverter>::get_pytype() [with ResultConverter = boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >; PyTypeObject = _typeobject]':
/opt/local/include/boost/python/detail/caller.hpp:240:19:   required from 'static boost::python::detail::py_func_sig_info boost::python::detail::caller_arity<1u>::impl<F, Policies, Sig>::signature() [with F = std::function<float(int)> (*)(const float*); Policies = boost::python::return_internal_reference<>; Sig = boost::mpl::vector2<std::function<float(int)>, const float*>]'
/opt/local/include/boost/python/object/py_function.hpp:48:35:   required from 'boost::python::detail::py_func_sig_info boost::python::objects::caller_py_function_impl<Caller>::signature() const [with Caller = boost::python::detail::caller<std::function<float(int)> (*)(const float*), boost::python::return_internal_reference<>, boost::mpl::vector2<std::function<float(int)>, const float*> >]'
Test.cc:14:1:   required from here
/opt/local/include/boost/python/detail/caller.hpp:102:109: error: 'struct boost::python::detail::reference_existing_object_requires_a_pointer_or_reference_return_type<std::function<float(int)> >' has no member named 'get_pytype'
         return create_result_converter((PyObject*)0, (ResultConverter *)0, (ResultConverter *)0).get_pytype();
                                                                                                             ^

Answer 1

Consider using the with_custodian_and_ward_postcall CallPolicy. This policy allows for the return types to be returned by value, while still extending the lifetime of another object to be at least as long as that of the returned object.

BOOST_PYTHON_MODULE(test) {
  def("func", func, with_custodian_and_ward_postcall<0, 1>());
}

---

As noted in the return_internal_reference documentation, the object returned references an existing internal object:

return_internal_reference [...] allow pointers and references to objects held internally [...] to be returned safely without making a copy of the referent.

The documentation also briefly mentions its use of with_custodian_and_ward_postcall. In summary, return_internal_reference has two note worthy effects on the function being exposed:

• The returned Python object neither has explicit nor shared ownership of the referenced C++ object.
• The returned Python object is a custodian, and will extend the lifetime of the ward object, indicated by the owner_arg, to be at least as long as the custodian.

As the returned Python object refers to an existing object for which it has neither explicit nor shared ownership, Boost.Python performs type checking to prevent creating a dangling reference. In the example code, func() returns a functor by value, resulting in a compiler error hinting that the return type must be either a pointer or reference:

struct boost::python::detail::
reference_existing_object_requires_a_pointer_or_reference_return_type

To explicitly control the lifetime of the returned object, one should consider using return_value_policy and models of ResultConverterGenerators. For example, if func() returned a functor by pointer that was created by new(), and wished to pass ownership of the object to Python while still maintaining the custodian and ward relationship, then one could chain policies with policy composition:

BOOST_PYTHON_MODULE(test) {
  def("func", func, 
    return_value_policy<manage_new_object,
      with_custodian_and_ward_postcall<0, 1> >());
}

---

Here is a complete minimal example based on the original code with verbose output to demonstrate the with_custodian_and_ward_postcall behavior:

#include <boost/python.hpp>
#include <iostream>

/// @brief Mockup class with verbose construction and destruction.
class foo
{
public:
  foo() { std::cout << "foo() " << this << std::endl; }
  foo(const foo&) { std::cout << "foo(const foo&) " << this << std::endl; }
  ~foo() { std::cout << "~foo() " << this << std::endl; }
};

/// @brief Mockup class with verbose construction and destruction.
class bar
{
public:
  bar() { std::cout << "bar() " << this << std::endl; }
  bar(const bar&) { std::cout << "bar(const bar&) " << this << std::endl; }
  ~bar() { std::cout << "~bar() " << this << std::endl; }
};

/// @brief Mockup factory function.
foo make_foo(bar& /* unused */)
{
  return foo();
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  // Do not allow Foo to be explicitly created from its type.
  python::class_<foo>("Foo", python::no_init);
  python::class_<bar>("Bar", python::init<>());

  // Expose make_foo, that returns a foo object when provided a
  // bar object.  The bar object's lifetime will be extended to
  // be at least as long as that of the returned foo object.
  python::def("make_foo", &make_foo,
    python::with_custodian_and_ward_postcall<
      0, // custodian = returned Foo object
      1  // ward = provided Bar object
    >());
}

Interactive usage:

>>> import example
>>> bar = example.Bar()
bar() 0x125ac30
>>> foo = example.make_foo(bar)
foo() 0x7fffa9b5efff
foo(const foo&) 0x7f1fcbe40090
~foo() 0x7fffa9b5efff
>>> bar = None
>>> foo = None
~foo() 0x7f1fcbe40090
~bar() 0x125ac30

Notice that despite the bar variable being set to None, the actual ward object remains alive until the foo custodian is destroyed.