How could I define the DoFn in apache crunch having "void" data type?

272 Views Asked by At

Basically, I don't need output from DoFn, just want to update some mysql db for each record I am getting in DoFn. So how could I define DoFn having void data type ? Basically I don't want to emit anything form DoFn.

1

There are 1 best solutions below

0
On

it is not possible. Once Crunch generate the graph for the execution, it determines if there is or not an output to be written or data to be materialized (pretty similar to a action). Even to accomplish what you want, you will need to do a workaround like create a DoFn<T,String> are code for write or materialize the PCollection result from your DoFn, even if you already know that there is not going to be any outputs.