I am utilizing AWS Glue jobs and want to utilize the job bookmarking feature. Right now, I checked the 'enable job bookmarking' box within AWS. However, I use the same Glue job when passing different input parameters for the job to run on different source data. I've encountered this problem when running into this error here when the job was running simultaneously with different parameters:
An error occurred while calling z:com.amazonaws.services.glue.util.Job.commit. Continuation update failed due to version mismatch. Expected version 21 but found version 23 (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: VersionMismatchException; Request ID: (my req id); Proxy: null)
I found something similar on stack overflow here (com.amazonaws.services.gluejobexecutor.model.VersionMismatchException)
But, I could not find a sufficient solution in this post. Has anyone else found a workaround to specify a single bookmark for a set of job parameters? Any help would be great. Creating separate Glue jobs will be a nightmare for code maintenance.
Tried, Expectation, Actual Result I tried running Glue jobs simultaneously with different input parameters with the job bookmarking feature enabled in AWS Glue.
I expected a separate job bookmark for each set of job parameters.
The actual result was a single job bookmark independent of different job parameters.