Bazel genrule python import from a script in different directory

Question

I have a Bazel native.genrule that calls a python script to generate a file. In the working configuration my files are the following:

Python script:

## test/sub1/test_python_script.py
def test_python_import():
    print("this is a function to test function import")


if __name__=="__main__":
    test_python_import()

BUILD file

## test/sub1/BUILD
load(":test.bzl", "test_python_import")

filegroup(
    name = "test_python_import_script",
    srcs = ["test_python_import.py"],
    visibility = ["//visibility:public"],
)

test_python_import(
    name = "test_python_import",
    out_ = "test.txt",
    visibility = ["//visibility:public"],
)

test.bzl file:

# test/sub1/test.bzl
def test_python_import(
        name,
        out_ = "",
        **kwargs):
    native.genrule(
        name = name,
        outs=[out_,],
        cmd = "python3 $(location //project/test/sub1:test_python_import_script) > $(@)",
        tools = ["//project/test/sub1:test_python_import_script:test_python_import_script"],
    )

From this working status, I want to move the function test_python_import() to another python file, so that it can be used by other scripts as well.

If I create the new file (common.py) under the same subfolder sub1, I can import the moved functions into the script test_python_import with:

from common import test_python_import

and everything works with no need to modify anything.

If the new script containing the function definition is in another folder i.e., /test/sub2/common.py:

# test/sub2/common.py
def test_python_import():
    print("this is a function to test function import")

and change the original script to include this function:

## test/sub1/test_python_script.py
from project.test.sub2 import test_python_import

if __name__=="__main__":
    test_python_import()

it does not work. I have tried to add a test/sub2/BUILD file to use filegroup and add this target to the tools and/or srcs attributes of the bazel macro:

# test/sub2/BUILD
filegroup(
    name = "common_script",
    srcs = ["common.py"],
    visibility = visibility = ["//visibility:public"],
)

New macro definition

# test/sub1/test.bzl
def test_python_import(
        name,
        out_ = "",
        **kwargs):
    native.genrule(
        name = name,
        outs=[out_,],
        srcs = ["//app/test/sub2:common_script",],
        cmd = "python3 $(location //app/test/sub1:test_python_import_script) > $(@)",
        tools = [
            "//app/test/sub1:test_python_import_script",
            "//app/test/sub2:common_script",
        ],
    )

but I get the error

Traceback (most recent call last):
  File "app/test/sub1/test_python_import.py", line 1, in <module>
    from app.test.sub2 import test_python_import
ModuleNotFoundError: No module named 'app'

I have tried to modify the test/sub2/BUILD file to define a python library by adding an empty __init__.py script and modifying the BUILD file to the following:

# test/sub2/BUILD
py_library(
    name = "general_script",
    deps = [],
    srcs = ["general_script.py"],
    visibility = ["//visibility:public"],
)

filegroup(
    name = "general_script",
    srcs = ["codegen_utils.py"],
    visibility = ["//visibility:public"],
)
filegroup(
    name = "init",
    srcs = ["__init__.py"],
    visibility = ["//visibility:public"],
)

and also to modify the genrule in different ways by adding both to srcs and tools fields the targets defined in the new app/test/sub2/common and then import in the test_python_import.py script the function, but whatever I've tried so far, I keep getting the same error.

How can import functions that are defined in a script into another script that lies into another directory within a genrule?

Answer 1

Rather than manually calling python3 myscript.py, it's better to specify a binary in genrule.tools and invoke that. In this case, use py_binary. If you want a separately usable library, put it in py_library and have the binary depend on it. Note you may need to set the py_library.imports attribute to get the proper paths onto sys.path to import it under the desired name.

genrule(
  cmd = "$(rootpath :mybin) > $(@)"
  tools = [":mybin"]
)
py_binary(
  name = "mybin",
  srcs = ["mybin.py"],
  deps = [":mylib"],
)
py_library(name = "mylib", srcs=["mylib.py"])

Note that this is generally true for any tool you're running in a genrule -- it's better to use the language-specific binary rules. This allows them to assemble the sources into a proper executable. It also reduces a source of system-specific dependencies. e.g., in order for cmd = "python3 bla.py" to work, it requires that the host system running the command have a compatible version of python3 available on the path in the genrule's shell environment (plus whatever other dependencies like third party libraries, etc). When going through e.g. py_binary, those dependencies are handled by the rule and Bazel's infrastructure.