How to prevent Python from throwing back to C++ boost::python::error_already_set?

Question

I need to give some context before the question. Please bear with me. Using boost::python I expose some exception types to Python, say MyExceptionType. I have a boomTest that I expose to Python to check that it works. Python calls boomTest and it handles MyExceptionType correctly, so far so good. This is the C++ side of things:

static void boomTest() {
    throw MyExceptionType("Smoked too many Cohibas!");
}

static PyObject *myExceptionPtr = NULL;
static void translate(MyExceptionType const &exception) {
    assert(myExceptionPtr != NULL);
    boost::python::object pythonExceptionInstance(exception);
    PyErr_SetObject(myExceptionPtr, pythonExceptionInstance.ptr());
}

BOOST_PYTHON_MODULE(test) {
    class_<MyExceptionType> myException("MyExceptionType", no_init);
    myException.add_property("message", &MyExceptionType::what);
    myExceptionPtr = myException.ptr();
    register_exception_translator<MyExceptionType>(&translate);
}

And this is the Python side of things:

import sys

import example
reload(example)
from example import MyExceptionType, boomTest

def tryBoomTest():
    try:
        boomTest()

    except MyExceptionType as ex:
        print 'Success! MyExceptionType gracefully handled:' \
            '\n message="%s"' % ex.message
    except:
        print 'Caught unhandled exception: %s "%s"' % (sys.exc_info()[0], sys.exc_info()[1])

Now things get a bit hairy because in a real use case, I have a callback to Python from a C++ Boost (non-Python) thread as follows:

# this is a Python callback invoked from a C++ boost non-Python thread
def handle(future):
    try:
        # future.get() throws MyExceptionType if there was a cluster exception
        "Cluster response received with value: %s" % future.get()

    except MyExceptionType as ex:
        print 'Success! MyExceptionType gracefully handled:' \
            '\n message="%s"' % ex.message

And now the OP:

Why is my C++ callback trigger getting a boost::python::error_already_set exception when the future.get() call throws a MyExceptionType intended for Python to handle? I suspect this behavior is caused by the fact that the exception is thrown within a C++ (non-Python) thread ...

What is needed to force Python handling the exception as in the example case at the beginning?

I have tried in the callback trigger from C++ to do the following:

void callbackTrigger() {
    try {
        pythonHandle_(getFuture());
    }
    // why do I get this???
    catch (boost::python::error_already_set&) {
        // this doesn't help, Python doesn't still handle MyExceptionType
        boost::python::handle_exception();
    }
}

Answer 1

I validated my theory about Python not liking to handle exceptions thrown (even in Python code) but executed as part of an alien C++ thread. Therefore I built this pure Python wrapper to be able to handle the callback in the main Python thread.

import example
reload(example)
from example import MyExceptionType

condition = threading.Condition()
futures = []

# Actual handle executed by the main Python THREAD
# __after__ submitting all the jobs to the Cluster
# N is the number of jobs that were submitted to the Cluster
def handle(N):
    while (N > 0):
        condition.acquire()
        try:
            # wait for max of a second to prevent this thread waiting indefinitely
            # when it misses notify while being busy processing responses
            condition.wait(1.0)
            for future in futures:
                try:
                    print 'callback received! the response is:\n %s' % future.get()

                except MyExceptionType as ex:
                    print 'MyExceptionType gracefully handled:' \
                          '\n message="%s"' % ex.message
                except:
                    print 'Caught unhandled exception: %s "%s"' % (sys.exc_info()[0], sys.exc_info()[1])
        finally:
            N -= len(futures)
            del(futures[:])
            condition.release()

# callback called from a C++ boost THREAD
def callback(future):
    condition.acquire()
    try:
        # do not consume the future here, rather let the main 
        # Python thread deal with it ...
        futures.append(future)
        condition.notify()
    finally:
        condition.release()

It is not ideal but it works and the output is correct:

registering callback 1 ...
registering callback 2 ...
registering callback 3 ...
registering callback 4 ...
registering callback 5 ...
MyExceptionType gracefully handled:
 message="Smoked too many Cohibas!"
MyExceptionType gracefully handled:
 message="Smoked too many Cohibas!"
MyExceptionType gracefully handled:
 message="Smoked too many Cohibas!"
MyExceptionType gracefully handled:
 message="Smoked too many Cohibas!"
MyExceptionType gracefully handled:
 message="Smoked too many Cohibas!"

Process finished with exit code 0