How do I connect Amazon mq to AWS emr serveless?

15 Views Asked by At

I have a Serverless EMR appication, I am submitting a spark job via python script. I have packaged all the dependencies an an the script to an s3 bucket. When I execute the job the spark job is running fine in fact all the dependencies are also getting imported for e.g. PIKA. but the issue is when I am trying to connect to the Rabbit mq via pika it gives the following error.

 Traceback (most recent call last):
  File "/tmp/spark-f27fd95b-3ec7-46b9-93f0-d9411a0e4a52/test_spark.py", line 108, in rabbit_mq_connection
    rabbit_mq_conn= pika.BlockingConnection(parameters)
  File "/home/hadoop/pyspark_deps/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/home/hadoop/pyspark_deps/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError

I have check the following:

  1. The server less executor has all the Amazon MQ full access
  2. Rabbit mq is publicly accessible
  3. I am able to connect to the rabbit mq instance via my lamda function

are there any permission missing ?

0

There are 0 best solutions below