GRPC - Separate proto serialization cpu stage from IO stage

Question

We have a service which sends grpc requests under heavy load. Normally, grpc send is considered to be an IO stage, which means it should use an IO thread pool. We've noticed that the serialization time of the proto objects consumes rather a-lot of cpu, which is not suitable to use with an IO thread pool. Is there some way to separate the serialization step from the IO step, e.g can we somehow do something like:

val bytes = myProto.toBytes()
myService.send(bytes)

We have no control over the .proto files and the server, but we can generate a client and a stub. We currently use scalapb but java might do fine as well.

Another option is to somehow translate the client calls to http2 calls, anyone ever tried that?

Any other suggestions on how to tackle this issue? Thanks in advance.

Answer 1

gRPC itself separates protobuf encoding from transmission. Protobuf encoding and decoding is performed on application threads: the threads calling gRPC and the threads delivering callbacks (e.g., channelBuilder.executor()). Then the I/O is processed asynchronously using separate network threads (Netty event loops). Just interact with gRPC on threads that can handle the protobuf CPU cost.