Is there a way to pass flags to runpy?

Question

I am looking for a way to pass miscellaneous options using runpy or other tools.

In particular, I would like to get the output of an optimized python script in another non-optimized python script.

python -O tobeoptimized.py

I have tried using subprocess but I can't extract the object that I need as I do in runpy.

from subprocess import PIPE, run
command = ['python','-O','tobeoptimized.py']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)

Answer 1

It's doable.
The compile built-in function takes optimize parameter. Its values are 0, 1 and 2. If it's 0 which is like without -O, 1 is -O and 2 is -OO on the command line.
To make runpy run module / path optimized it must be patched. Before doing that I'll define two functions for illustration.

This is a test module. If it's run without optimization it won't print.
app.py

assert 0, "assert zero"
print("Optimized")

These two functions don't do all the details as runpy does. So won't function in full.

import importlib
import runpy

optimize = 1


def run_module(mod_name):
    spec = importlib.util.find_spec(mod_name)
    source = spec.loader.get_source(spec.name)
    code = compile(source, spec.name + ".py", "exec", optimize=optimize)
    d = {}
    exec(code, d)
    return d


def run_path(path):
    with open(path) as f:
        source = f.read()
    code = compile(source, path, "exec", optimize=optimize)
    d = {}
    exec(code, d)
    return d

This one is to patch a function in runpy which runpy.run_module uses to get code object of the module to run. The patch provides optimized code object.

import runpy

optimize = 1

def _get_module_details_wrapper(func):
    def tmp(*args, **kwargs):
        mod_name, spec, _ = func(*args, **kwargs)
        source = spec.loader.get_source(spec.name)
        code = compile(source, spec.name + ".py", "exec", optimize=optimize)
        return mod_name, spec, code

    return tmp

runpy._get_module_details = _get_module_details_wrapper(runpy._get_module_details)
runpy.run_module('app')

UPDATE
runpy.run_path can be run with optimization turned on.

def optimize_compile(o):
    from functools import partial
    import builtins

    builtins.compile = partial(compile, optimize=o)


optimize_compile(1)
runpy.run_path("app.py")


optimize_compile(0)
try:
    runpy.run_path("app.py")
except AssertionError:
    print("assertion")

optimize_compile(1)
runpy.run_path("app.py")

Answer 2

Not with runpy because it isn't spawning a new process, (exec()). If it could have been done this way, this question would have had an answer. So we're pretty much falling into the C codebase or extension world where the core of CPython is modified in such a manner it can be accessed at least from C API if not from Python API.

Instead try using subprocess for passing the flags. The output can be adjusted.

If the problem is that you want to extract an object out of the process (thus subprocess "failing") you need to look into the pickle module or simply drop it to a file/buffer/socket and retrieve the data + reassemble in a custom manner.

Subprocess isn't really mandatory, however if there were any other implementation, it won't "hack it away" because it's the compiled core that's in the way, not something that can be monkey-patched away.

I guess we need a new feature from the core devs? :)

Regarding the subprocess output, this works for me just fine.

# hello.py
print("Hello")

# main.py
from subprocess import Popen, PIPE

process = Popen(
    ["python", "-O", "hello.py"],
    stdout=PIPE, stderr=PIPE, universal_newlines=True
)
print(process.stdout.read())