Achieve relative path in generated module imports with grpcio-tools

Question

I am generating modules from two different protos, which have some common proto file names. From protos-1 ->

python3 -m grpc_tools.protoc --proto_path=p1/protos/ --python_out=. --grpc_python_out=. p1/protos/common/*.proto
python3 -m grpc_tools.protoc --proto_path=p1/protos/ --python_out=. --grpc_python_out=. p1/protos/p1-unique/*.proto

From protos-2 ->

python3 -m grpc_tools.protoc --proto_path=p2/protos/ --python_out=. --grpc_python_out=. p2/protos/p2-unique/*.proto
python3 -m grpc_tools.protoc --proto_path=p2/protos/ --python_out=. --grpc_python_out=. p2/protos/common/*.proto

Running above commands for p1 followed by p2 overrides the generated module common of p1. Hence, it has issues in the generated file p1-unique_pb2.py. (incorrect ref of common module)

I then tried to generate modules from protos-2 in a separate folder, with

python3 -m grpc_tools.protoc --proto_path=p2/protos/ --python_out=./abc --grpc_python_out=./abc p2/protos/p2-unique/*.proto
python3 -m grpc_tools.protoc --proto_path=p2/protos/ --python_out=./abc --grpc_python_out=./abc p2/protos/common/*.proto

but still the imports in p2-unique_pb2.py is absolute from common import common_pb2 as common_dot_common__pb2 and hence referring to common module generated by protos-1

I tried several solutions provided.. https://github.com/grpc/grpc/issues/9575 but none helped in solving this issue.

Answer 1

This is technically a ProtoBuf issue. From the description, I don't know if the common/*.proto is the same across protos-1 and protos-2. If they are the same, I guess you won't be posting this issue at all. If they are different, you may try matching the folder structure with the proto package name.

For example, you have a proto file bar.proto with package name p2.common, then it should be placed at ./p2/common/bar.proto. The protoc command can just take proto_path=.. If there are multiple projects, you can consider copying all proto files into a central place before compilation.

There are many known complaint about the Python import path generation, you can find more info in https://github.com/protocolbuffers/protobuf/issues.