I'm working on simple beam dataflow in JAVA on google cloud platform. I've tested locally and the pipeline is running well.
When i deploy on dataflow, i got this looping error :
{
insertId: "10mkpdlb7i"
labels: {4}
logName: "projects/myproject/logs/dataflow.googleapis.com%2Fjob-message"
receiveTimestamp: "2022-11-14T09:09:08.216192037Z"
resource: {
labels: {
job_id: "2022-11-14_01_20_11-8382597753945077942"
job_name: "production-validation-pipeline"
project_id: "3456543"
region: "europe-west1"
step_id: ""
}
type: "dataflow_step"
}
severity: "ERROR"
textPayload: "Workflow failed."
timestamp: "2022-11-14T09:09:07.793137682Z"
}
My dataflow :
- read from kafka
- filter message -> keep some of them based on value in the message
- convert to pubsub message
- push to pubsub
Thanks in advance for your help