Google Dataflow with "Workflow failed"

345 Views Asked by At

I'm working on simple beam dataflow in JAVA on google cloud platform. I've tested locally and the pipeline is running well.

When i deploy on dataflow, i got this looping error :

{
  insertId: "10mkpdlb7i"
  labels: {4}
  logName: "projects/myproject/logs/dataflow.googleapis.com%2Fjob-message"
  receiveTimestamp: "2022-11-14T09:09:08.216192037Z"
  resource: {
    labels: {
      job_id: "2022-11-14_01_20_11-8382597753945077942"
      job_name: "production-validation-pipeline"
      project_id: "3456543"
      region: "europe-west1"
      step_id: ""
    }
    type: "dataflow_step"
  }
  severity: "ERROR"
  textPayload: "Workflow failed."
  timestamp: "2022-11-14T09:09:07.793137682Z"
}

My dataflow :

  • read from kafka
  • filter message -> keep some of them based on value in the message
  • convert to pubsub message
  • push to pubsub

Thanks in advance for your help

0

There are 0 best solutions below