IPython Notebook Javascript: retrieve content from JavaScript variables

Question

Is there a way for a function (called by an IPython Notebook cell) to retrieve the content of a JavaScript variable (for example IPython.notebook.notebook_path which contains the path of the current notebook)?

The following works well when written directly within a cell (for example, based on this question and its comments):

from IPython.display import display,Javascript
Javascript('IPython.notebook.kernel.execute("mypath = " + "\'"+IPython.notebook.notebook_path+"\'");')

But that falls apart if I try to put it in a function:

# this doesn't work
from IPython.display import display,Javascript
def getname():
    my_js = """
    IPython.notebook.kernel.execute("mypath = " + "\'"+IPython.notebook.notebook_path+"\'");
    """
    Javascript(my_js)
    return mypath

(And yes, I've tried to make global the mypath variable, both from within the my_js script and from within the function. Also note: don't be fooled by possible leftover values in variables from previous commands; to make sure, use mypath = None; del mypath to reset the variable before calling the function, or restart the kernel.)

Another way to formulate the question is: "what's the scope (time and place) of a variable set by IPython.notebook.kernel.execute()"?

I think it isn't an innocuous question, and is probably related to the mechanism that IPython uses to control its kernels and their variables and that I don't know much about. The following experiment illustrate some aspect of that mechanism. The following works when done in two separate cells, but doesn't work if the two cells are merged:

Cell [1]:

my_out = None
del my_out
my_js = """
IPython.notebook.kernel.execute("my_out = 'hello world'");
"""
Javascript(my_js)

Cell [2]:

print(my_out)

This works and produces the expected hello world. But if you merge the two cells, it doesn't work (NameError: name 'my_out' is not defined).

Answer 1

Okay, I found a way around the problem: call a Python function from Javascript and have it do all of what I need, rather than returning the name to "above" and work with that name there.

For context: my colleagues and I have many experimental notebooks; we experiment for a while and try various things (in a machine learning context). At the end of each variation/run, I want to save the notebook, copy it under a name that reflects the time, upload it to S3, strip it from its output and push it to git, log the filename, comments, and result scores into a DB, etc. In short, I want to automatically keep track of all of our experiments.

This is what I have so far. At the bottom of my notebooks, I put:

In [127]: import mymodule.utils.lognote as lognote
          lognote.snap()

In [128]: # not to be run in the same shot as above
          lognote.last
Out[128]: {'file': '/data/notebook-snapshots/2015/06/18/20150618-004408-save-note-exp.ipynb',
           'time': datetime.datetime(2015, 6, 18, 0, 44, 8, 419907)}

And in a separate file, e.g. mymodule/utils/lognote.py:

# (...)

from datetime import datetime
from subprocess import call
from os.path import basename, join
from IPython.display import display, Javascript

# TODO: find out where the topdir really is instead of hardcoding it
_notebook_dir = '/data/notebook'
_snapshot_dir = '/data/notebook-snapshots'

def jss():
    return """
    IPython.notebook.save_notebook();
    IPython.notebook.kernel.execute("import mymodule.utils.lognote as lognote");
    IPython.notebook.kernel.execute("lognote._snap('" + IPython.notebook.notebook_path + "')");
    """
def js():
    return Javascript(jss())

def _snap(x):
    global last
    snaptime = datetime.now()
    src = join(_notebook_dir, x)
    dstdir = join(_snapshot_dir, '{}'.format(snaptime.strftime("%Y/%m/%d")))
    dstfile = join(dstdir, '{}-{}'.format(snaptime.strftime("%Y%m%d-%H%M%S"), basename(x)))
    call(["mkdir", "-p", dstdir])
    call(["cp", src, dstfile])
    last = {
        'time': snaptime,
        'file': dstfile
        }

def snap():
    display(js())

Answer 2

To add to the other great answers, there is a nuance of the browsers attempting to run the jupyter nb javascript magic on nb load.

To demonstrate: create and run the following cell:

%%javascript 
IPython.notebook.kernel.execute('1')

Now save the notebook, close it and then re-open it. When you do that, under that cell suddenly you will see an error in red:

Javascript error adding output!
TypeError: Cannot read property 'execute' of null
See your browser Javascript console for more details.

That means the browser has parsed some js code and it tried to run it. This is the error in chrome, it will probably different in a different browser.

I have no idea why this jupyter javascript magic cell is being run on load and why jupyter notebook is not properly escaping things, but the browser sees some js code and so it runs it and it fails, because the notebook kernel doesn't yet exist!

So you must add a check that the object exists:

%%javascript 
if (IPython.notebook.kernel) {
    IPython.notebook.kernel.execute('1')
}

and now there is no problem on load.

In my case, I needed to save the notebook and run an external script on it, so I ended up using this code:

from IPython.display import display, Javascript
def nb_auto_export():
    display(Javascript("if (IPython.notebook) { IPython.notebook.save_notebook() }; if (IPython.notebook.kernel) { IPython.notebook.kernel.execute('!./notebook2script.py  ' + IPython.notebook.notebook_name )}"))

and in the last cell of the notebook:

nb_auto_export()

Answer 3

I think the problem is related with Javascript being asynchronus while python is not. Normally you would think that the Javascript(""" python cmd """) command is executed, and then your print statment should work properly as expected. However, the Javascript command is fired but not executed. Most pobably it is executed after the cell 1 execution is fully completed.

I tried your example with sleep function. Did not help.

The asnyc problem can esaily be seen by adding an alert statement within my_js, but before kernel.execute line. The alert should be fired even before trying a python command execution.

But at the presence of print (my_out) statement within cell 1, you will again get the same error without any alerts. If you take the print line out, you will see the alert poping out within cell 1. But the varibale my_out is set afterwards.

my_out = None
del my_out
my_js = """
**alert ("about to execute python comand");**
IPython.notebook.kernel.execute("my_out = 'hello world'");
"""
Javascript(my_js)

There are other javascript utilities within notebook like IPython.display.display_xxx which varies from displaying video to text object, but even the text object option does not work.

Funny enough, I tested this with my webgl canvas application which displays objects on the HTML5 canvas; display.display_javascript(javascript object) works fine ( which is a looong html5 document) while the two pieces of words of output does not show up?! Maybe I should embed the output into canvas application somewhere, so it s displayed on the canvas :)

Answer 4

I wrote a related question (Cannot get Jupyter notebook to access javascript variables) and came up with a hack that does the job. It uses the fact that the input(prompt) command in Python does block the execution loop and waits for user input. So I looked how this is processed on the Javascript side and inserted interception code there.

The interception code is:

import json
from IPython.display import display, Javascript
display(Javascript("""
  const CodeCell = window.IPython.CodeCell;

  CodeCell.prototype.native_handle_input_request = CodeCell.prototype.native_handle_input_request || CodeCell.prototype._handle_input_request
  CodeCell.prototype._handle_input_request = function(msg) {
    try {
      // only apply the hack if the command is valid JSON
      console.log(msg.content.prompt)
      const command = JSON.parse(msg.content.prompt);
      const kernel = IPython.notebook.kernel;
      // return some value in the Javascript domain, depending on the 'command'.
      // for now: specify a 5 second delay and return 'RESPONSE'
      kernel.send_input_reply(eval(command["eval"]))
    } catch(err) {
      console.log('Not a command',msg,err);
      this.native_handle_input_request(msg);
    }
  }
"""))

The interception code looks whether the input prompt is valid JSON, and in that case it executes an action depending on the command argument. In this case, it runs the commend["eval"] javascript expression and returns the result. After running this cell, you can use:

notebook_path = input(json.dumps({"eval":"IPython.notebook.notebook_path"}))

Quite a hack, I must admit.