Spring boot/spring-cloud-aws SQS failing to poll messages when deployed to AWS/EC2 environment

Question

Spring boot app works fine running locally connecting to sandbox S3 & sandbox SQS, using DefaultAWSCredentialsProviderChain and set as system property.

When application is deployed to EC2 environment and using ProfileCredentials, I get a continuous stream of following error in CloudWatch:

{
    "Host": "<myhost>",
    "Date": "2016-12-20T21:52:56,777",
    "Thread": "simpleMessageListenerContainer-1",
    "Level": "WARN ",
    "Logger": "org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer",
    "Msg": "An Exception occurred while polling queue 'my-queue-name'. The failing operation will be retried in 10000 milliseconds",
    "Identifiers": {
        "Jvm-Instance": "",
        "App-Name": "my-app",
        "Correlation-Id": "ca9a556e-2fbc-3g49-9fb8-0e9213bb79bc",
        "Session-Id": "",
        "Thread-Group": "main",
        "Thread-Id": "32",
        "Version": ""
    }
}
java.lang.NullPointerException
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$AsynchronousMessageListener.run(SimpleMessageListenerContainer.java:255) [spring-cloud-aws-messaging-1.1.1.RELEASE.jar:1.1.1.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]

The problem boils down to SimpleMessageListenerContainer.java:255 :

ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());

this.queueAttributes is null.

I have tried everything, including @EnableContextCredentials(instanceProfile=true), to setting cloud.aws.credentials.instanceProfile=true while making sure access & secretKey is null. The SQS queue definitely exists, and I have verified through aws cli on the EC2 instance itself that the profile credentials exist and are valid.

Additionally, in AWS environment the app also used S3 client to generate unique keys for bucket storage, which all works. It's only when the app tries to poll messages from SQS that seems to be failing.

I am processing messages like so:

@SqsListener("${aws.sqs.queue.name}")
public void receive(S3EventNotification s3EventNotificationRecord) {

more config:

@Bean
   public AWSCredentialsProvider awsCredentialsProvider(
           @Value("${aws.credentials.accessKey}") String accessKey,
           @Value("${aws.credentials.secretKey}") String secretKey,
           JasyptPropertyDecryptor propertyDecryptor) {
       if (!Strings.isNullOrEmpty(accessKey) || !Strings.isNullOrEmpty(secretKey)) {
           Preconditions.checkState(
                   !Strings.isNullOrEmpty(accessKey) && !Strings.isNullOrEmpty(secretKey),
                   "Error in accessKey/secretKey config. Either both must be provided, or neither.");
           System.setProperty("aws.accessKeyId", propertyDecryptor.decrypt(accessKey));
           System.setProperty("aws.secretKey", propertyDecryptor.decrypt(secretKey));
       }
       return DefaultAWSCredentialsProviderChain.getInstance();
   }

   @Bean
   public S3Client s3Client(
           AWSCredentialsProvider awsCredentialsProvider,
           @Value("${aws.s3.region.name}") String regionName,
           @Value("${aws.s3.bucket.name}") String bucketName) {
       return new S3Client(awsCredentialsProvider, regionName, bucketName);
   }

   @Bean
   public QueueMessageHandlerFactory queueMessageHandlerFactory() {

       MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();
       messageConverter.setStrictContentTypeMatch(false);

       QueueMessageHandlerFactory factory = new QueueMessageHandlerFactory();
       factory.setArgumentResolvers(
           Collections.<HandlerMethodArgumentResolver>singletonList(
               new PayloadArgumentResolver(messageConverter)));

       return factory;
   }

One additional thing I noticed is that on application start up, ContextConfigurationUtils.registerCredentialsProvider is called and unless you specify cloud.aws.credentials.profileName= as empty in your app.properties, this class will add a ProfileCredentialsProvider to the list of awsCredentialsProviders. I figured this might be problematic since I'm not providing credentials on the EC2 instance that way, and instead it should be using InstanceProfileCredentialsProvider. This change did not work.

Answer 1

Turns out the issue was that the services I was using in AWS such as SQS has proper access permissions on them, but the IAM profile itself lacked the permissions to even attempt the service operations that the application needed to make.