I'm running a Google Cloud Dataproc Serverless Spark batch workload with Runtime version 2.1
The batch is getting Cancelled after Running for 4 hours
As per Google Cloud Dataproc Serverless Guides https://cloud.google.com/dataproc-serverless/docs/quickstarts/spark-batch#submit_a_spark_batch_workload
--ttl: You can add the --ttl flag to specify the duration of the batch lifetime. When the workload exceeds this duration, it is unconditionally terminated without waiting for ongoing work to finish. Specify the duration using a s, m, h, or d (seconds, minutes, hours, or days) suffix. The minimum value is 10 minutes (10m), and the maximum value is 14 days (14d).
1.1 or 2.0 runtime batches: If --ttl is not specified for a 1.1 or 2.0 runtime batch workload, the workload is allowed to run until it exits naturally (or run forever if it does not exit).
2.1+ runtime batches: If --ttl is not specified for a 2.1 or later runtime batch workload, it defaults to 4h.
I tried adding --ttl flag, but getting an error
ERROR: (gcloud.dataproc.batches.submit.spark) unrecognized arguments: --ttl=6h
Example
gcloud dataproc batches submit spark --version=2.1 --region=us-east4 --subnet=$subnet --service-account=$service_account --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar --class=org.apache.spark.examples.SparkPi --ttl=6h -- 1000
I'm expecting to specify the duration of the batch lifetime.
Answering my own question
After @John Hanley comment (thanks for the tip!), I checked gcloud version which is Google Cloud SDK 383.0.1 alpha 2022.04.26 beta 2022.04.26 bq 2.0.74 bundled-python3-unix 3.8.11 core 2022.04.26 gsutil 5.9
I tried running the batch in another terminal with gcloud version Google Cloud SDK 418.0.0 alpha 2023.02.13 beta 2023.02.13 bq 2.0.85 bundled-python3-unix 3.9.16 core 2023.02.13 gcloud-crc32c 1.0.0 gsutil 5.20 and it worked this time.
I think --ttl flag was added in more recent gcloud versions.