I have a Serverless EMR appication, I am submitting a spark job via python script. I have packaged all the dependencies an an the script to an s3 bucket. When I execute the job the spark job is running fine in fact all the dependencies are also getting imported for e.g. PIKA. but the issue is when I am trying to connect to the Rabbit mq via pika it gives the following error.
Traceback (most recent call last):
File "/tmp/spark-f27fd95b-3ec7-46b9-93f0-d9411a0e4a52/test_spark.py", line 108, in rabbit_mq_connection
rabbit_mq_conn= pika.BlockingConnection(parameters)
File "/home/hadoop/pyspark_deps/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/home/hadoop/pyspark_deps/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
I have check the following:
- The server less executor has all the Amazon MQ full access
- Rabbit mq is publicly accessible
- I am able to connect to the rabbit mq instance via my lamda function
are there any permission missing ?