I'm looking into terratest for testing aws code. My workload mainly uses glue python or glue spark to access data from some s3, do some work, store to s3.
I want to write a test that tests whether my role correctly gives write access to some bucket over glue. For that purpose I create a glue pythonshell job which puts a file into the bucket, and then I would check (from my go code) wether the file and its contents are what is expected.
When trying to implement this the Terraform part was quickly written. I'm now stuck on how to trigger the python job and how to wait for it to finish.
Is there any decent way to do this?
Glue has triggers concept which helps you to start them based on the requirement.A trigger fires on demand, based on a schedule, or based on a combination of events.
Also once the job is started you can monitor the status of it using CloudWatch events. This article talks about how you can monitor a Glue crawler run.The same can be applied to Glue job with changes.