How to identify which logstream belongs which glue job?

Question

I have AWS Glue jobs of type python shell. Logs of the python shell glue jobs are getting stored in two different cloudwatch log groups.

1. /aws-glue/python-jobs/output:- To store non error logs
2. /aws-glue/python-jobs/error:- To store Error logs.

in each run of the glue jobs, new log streams get created under the above log groups. I was unable to identify which log stream belongs to which glue job. Because in each run a new log stream gets created. How to identify a glue job from the cloud watch log group log stream

Answer 1

When the Glue job run creates the log streams, it uses a default naming convention with job run ID/executor ID.

Inside the output or error logs, you would see a prefix with Glue job run ID prefix along with executor numbers. That helps to identify which job run it belongs so.

Other than that, if you want, it allows to define a custom prefix using this property.

'--continuous-log-logStreamPrefix': 'custom_log_stream_prefix'

More details can be found here : Enabling Continuous Logging for AWS Glue Jobs