How do closures work in runpy?

Question

I get unexpected behaviour when I try to run methods defined in a file loaded using the runpy module. The methods do not see any variables (including imported modules) defined outside of that method. Here is how I am doing it:

#test.py
import runpy
env = runpy.run_path('test', {'y':'world'})
env['fn']()

~

#test
import re

print(re.compile(r'^hello', re.IGNORECASE).sub('', "hello world"))
x = "hello"
print(x)
print(y)

def fn():
    try:
        print(re.compile(r'^hello', re.IGNORECASE).sub('', "hello world"))
    except:
        print("No re")
    try:
        print(x)
    except:
        print("No x")
    try:
        print(y)
    except:
        print("No y")

My expected output of test.py would be:

 world
hello
world
 world
hello
world

because fn would form a closure for re, x and y.

However, instead I get:

 world
hello
world
No re
None
None

It looks like re isn't defined within fn even though it should be with normal closure behaviour. x and y are even stranger because they appear to be defined but set to None.

Why is this and how do closures work with runpy? How can I achieve normal behaviour such that fn can 'see' outside variables?

Answer 1

OK, this is a curiosity of the way Python handles modules, which I know about but don't fully understand. I've come across it while working on IPython, where it's explained in a comment.

When Python runs a module, it produces a module object, the attributes of which are the global names in the module. When the module falls out of scope and is being destroyed, these attributes are set to None. Code which was defined in the function then sees these as the globals, as you found. You can demonstrate this by adding def g(): return globals() to your file, then calling env["g"]().

I don't know if there's a way round this with runpy. IPython uses some complicated code to reuse a module object for running other files, caching copies of its __dict__ to keep the references therein alive. Have a look at the magic_run function if you're interested.