retrieving python variables from Cell Magic in Ipython notebook from background subprocess

Question

I'm trying to do a quick and dirty querying of my mongodb database using ipython notebook.

I have several cells each with its own query. Since mongodb can support several connections I would like to run each query in parallel. I thought an ideal way would be just do something like

%%script --bg python
query = pymongo.find(blahbalhba)

You can imagine several cells each with its own query. However I'm not able to access the query returned by pymongo.find.

I understand that this is a subprocess run in a seperate thread, but I have no idea how to access the data since the process is quickly destroyed and the namespace goes away.

I found a similar post for %%bash here but I'm having trouble translating this to a python namespace.

Answer 1

%%script is just a convenient magic, it will not replace writing a full blown magic. The only thing I can see is to write your own magic. Basically if you can do it with a function that takes a string a parameter, you know how to write a magic.

So How would you (like to) write it in pure python ? (Futures, multiprocessing, queuing library ?) ... then move it to a magic.