Implicit relative imports inside of Protobuf-generated Python package make importing from outside impossible

Question

I have a package that looks like this:

setup.py
requirements.txt
alphausblue/
    api/
        ripple/
            org_pb2.py
    org/
        v1/
            org_pb2_grpc.py

In org_pb2_grpc.py I have the following import line:

from api.ripple import org_pb2 as api_dot_ripple_dot_org__pb2

The problem I'm having is that, when I import from inside the alphausblue directory, the import works correctly. However, when I create a wheel from the package, upload it to Test PyPI, download it again and attempt to import it into a test environment like this:

>>> import alphausblue
>>> import alphausblue.org
>>> import alphausblue.org.v1
>>> import alphausblue.org.v1.org_pb2_grpc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\anaconda3\envs\test\lib\site-packages\alphausblue\org\v1\org_pb2_grpc.py", line 5, in <module>
    from api.ripple import org_pb2 as api_dot_ripple_dot_org__pb2
ModuleNotFoundError: No module named 'api'

I get the error above. I can move the code from alphausblue/ to / and then the import works but I have to do org.v1.org_pb2_grpc, which is not what I want. It seems that this should work so what am I missing here?

Update This appears to be an issue with how protoc generates the Python code. I will investigate potential solutions here and see if I can't solve the issue myself.

Answer 1

So, I tried a number of things in an attempt to get Protobuf to do what I want, i.e.. work, but was unable to figure it out. So, I made a modification to the build script to do this:

mkdir -p generated/py/alphausblue
python3 -m grpc_tools.protoc -I . --python_out=./generated/py/alphausblue --grpc_python_out=./generated/py/alphausblue \
        ./org/v1/*.proto \
        ./kvstore/v1/*.proto \
        ./iam/v1/*.proto \
        ./admin/v1/*.proto \
        ./cost/v1/*.proto \
        ./billing/v1/*.proto \
        ./operations/v1/*.proto \
        ./preferences/v1/*.proto

python3 -m grpc_tools.protoc -I . --python_out=./generated/py/alphausblue \
        $(for v in $(find ./api -type d); do echo -n "$v/*.proto "; done)

for package in $(find generated/py/alphausblue -mindepth 1 -maxdepth 1 -type d -printf "%f "); do
        echo "Found package ${package}. Beginning replacement"
        find generated/py/alphausblue/. -name '*.py' -exec sed -i -e "s/from ${package}/from alphausblue.${package}/g" {} \;
done

The first two commands here generates the Python code from the .proto files. The next command uses find and sed to get all the top-level package names in the package itself and replace the implicit relative imports with absolute imports.

Answer 2

I think this can help:

import os
import re

# Set the path to the directory containing the generated files
generated_files_dir = ""

# Set the relative import prefix
relative_import_prefix = "."

# Get a list of all generated files in the directory
generated_files = [file for file in os.listdir(generated_files_dir) if "pb2" in file]

# Define the regular expression pattern to match import statements
import_pattern = re.compile(r"^import\s+(\S+)\s+as\s+(\S+)\s*$", re.MULTILINE)

# Iterate over each generated file and modify the import statements
for generated_file in generated_files:
    generated_file_path = os.path.join(generated_files_dir, generated_file)

    # Read the contents of the generated file
    with open(generated_file_path, "r") as f:
        file_contents = f.read()

    # Modify the import statements using relative imports
    modified_contents = import_pattern.sub(r"from {} import \1 as \2".format(relative_import_prefix), file_contents)

    # Write the modified contents back to the generated file
    with open(generated_file_path, "w") as f:
        f.write(modified_contents)