How can I increase the thread stack size on Apache Beam pipeline workers with Google Cloud Dataflow?

Question

I'm getting a StackOverflowError on my Beam workers due to running out the thread stack, and because it's deep within the running of a SqlTransform it's not straightforward to reduce the number of calls being made.

Is it possible to change the JVM thread stack size for my workers, either through Google Cloud Dataflow or Beam's own pipeline options?

Answer 1

I don't think there's an easy way to do this.

• If this is an issue of stack trace being purged by Cloud Logging, may be it might be possible to catch the exception yourself and inspect that instead of just logging it.
• If this is an issue of default stack trace depth set by the JVM not being enough, I don't think there's a way to update this for Dataflow today unfortunately.