I am trying to create a AWS Glue Job that will be triggered whenever a message is posted to a Kafka topic.
I created a Kafka cluster and topic using Confluent Portal. I was able to post messages to this topic using the Rest API endpoint provided. Once I created this I saw the Cluster Settings displayed. The cluster settings list a bootstrap server (host:9092) and Rest Endpoint which I used to send test messages.
Once this was created, I created a data connection in AWS Glue. I chose the connection type as "Kafka" and chose the "Customer managed Apache Kafka" option. Under the Kafka bootstrap URLs, I provided the bootstrao server that I saw under cluster settings in Confluent portal (host:9092). I unselected "Require SSL" and also set Authentication to None. I did not provide any values under Network Options.
I then proceeded to create a job from the data connection screen and selected the Kafka connection created and set the topic name. However when I run the job I see the below error -
jobname: KafkaJob and JobRunID:XXXX failed to execute wth exception Unable to resolve any valid service connection (Service AWSGlueJobExecutor, Status Code 400, Error Code InvalidInputExcepion)
Am I missing any steps to setup the Kafka data connection?