s3bucketnotifications aren't working with localstack and docker

Question

My situation is the following: There is an xml that gets uploaded to Amazon S3 via a pre-signed url. Once the file is uploaded on S3, we want an SQS queue to know via a S3 event notification. Once the SQS queue received this new event, we want a Lambda function to get triggered.

I am trying to get this flow running with a small POC with localstack in Docker. My Bucket, SQS queue and Lambda's (MyLambdaFunction and my bucket notifications handler) get created. Yet, when I do an effective upload to S3, the file doesn't go through the entire flow. Once It's on S3, nothing else happens. What am I doing wrong? I don't have the localstack PRO edition.

This is the code for my AWS CDK project: Inside my /bin folder I have a test.js file:

#!/usr/bin/env node
const cdk = require('aws-cdk-lib');
const { TestStack } = require('../lib/test-stack');
const app = new cdk.App();
new TestStack(app, 'TestStack');

Inside my /lib folder I have a test-stack.js file:

const cdk = require("aws-cdk-lib");
const sqs = require("aws-cdk-lib/aws-sqs");
const s3 = require("aws-cdk-lib/aws-s3");
const lambda = require("aws-cdk-lib/aws-lambda");
const s3Notifications = require("aws-cdk-lib/aws-s3-notifications");
const { SqsEventSource } = require("aws-cdk-lib/aws-lambda-event-sources");

class TestStack extends cdk.Stack {
  /**
   * @param {cdk.App} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    // S3 bucket
    const bucket = new s3.Bucket(this, "MyBucket", {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    // SQS queue
    const queue = new sqs.Queue(this, "MyQueue", {
      visibilityTimeout: cdk.Duration.seconds(300),
    });

    // Lambda function
    const lambdaFunction = new lambda.Function(this, "MyLambdaFunction", {
      runtime: lambda.Runtime.NODEJS_18_X,
      handler: "lambda.handler",
      code: lambda.Code.fromAsset("lib/lambda"),
      environment: {
        QUEUE_URL: queue.queueUrl,
      },
    });

    // Set up S3 event notification to trigger SQS
    bucket.addEventNotification(
      s3.EventType.OBJECT_CREATED,
      new s3Notifications.SqsDestination(queue),
    );

    // Set up SQS event source for Lambda function
    lambdaFunction.addEventSource(new SqsEventSource(queue));
  }
}

My docker-compose.yml file looks like:

version: "3.8"

services:
  localstack:
    image: localstack/localstack:3.2.0
    network_mode: bridge
    ports:
      - "4566:4566"
      - "4571:4571"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./localstack:/etc/localstack/init/ready.d

I execute the following in cmd: docker-compose up -d cdklocal bootstrap cdklocal deploy

Answer 1

This is a limitation of CDK, it creates a CustomResource for adding the S3 Notification to your bucket, which is a LocalStack Pro feature. You can use a workaround by accessing the L1 construct directly to set the configuration:

    const bucket = new s3.Bucket(...);
    const lambda = new NodejsFunction(...);
    if (isDeployingToLocalStack()) { // implement or replace `isDeployingToLocalStack()` as required
      const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
      cfnBucket.notificationConfiguration = {
        lambdaConfigurations: [
          {
            creationStack: [],
            event: 's3:ObjectCreated:*',
            function: lambda.functionArn,
          },
        ],
      };
    }

This solution was given on the LocalStack GitHub issue here: https://github.com/localstack/localstack/issues/9352#issuecomment-1862125662

Edit: following your comment, this would be the result for your case, setting up SQS notifications.

const bucket = new s3.Bucket(...);
const lambda = new NodejsFunction(...);
if (isDeployingToLocalStack()) {
  const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
  cfnBucket.notificationConfiguration = {
    queueConfigurations: [{
      event: s3.EventType.OBJECT_CREATED,
      queue: '<queue-arn>',
    }],
  };
}