How to handle IN clause with many elements in Calcite SQL within Apache BEAM

Question

I'm using Apache Beam version 2.29 and doing a sql transform. I have an IN clause in my sql with with many elements within (50+). When running I get the following error:

Exception in thread "main" java.lang.UnsupportedOperationException: CROSS JOIN, JOIN ON FALSE is not supported!

After some troubleshooting I found out that it's because IN only supports up to 19 elements (bug or feature?). I split my IN in smaller chunks with an OR but now I'm getting this error:

Caused by: org.codehaus.janino.InternalCompilerException: Code of method "eval0(Lorg/apache/beam/sdk/schemas/Schema;Lorg/apache/beam/sdk/transforms/DoFn$ProcessContext;Lorg/apache/beam/vendor/calcite/v1_20_0/org/apache/calcite/DataContext;)V" of class "SC" grows beyond 64 KB

Any idea what to do?

Answer 1

try search IN_SUB_QUERY_THRESHOLD, beyond IN_SUB_QUERY_THRESHOLD, in will be translated to join