Why am I not able to send curl request to Docker image when using aws-lambda-rie?

Question

Have spent the past day on this and truly cannot figure out why this is happening. Curl command to Docker container running aws lambda runtime emulator is hanging indefinitely and my CMD and ENTRYPOINTs are not running.

My handler is like the following (found in /src/main.py):

import logging
import json 
import httpx 
from utils import foo

logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.info("Set up logger") # Figured I'd just see if it's even getting here

var1 = ""
var2 = ""

def lambda_handler(event, context):
    global var1 
    global var2
    # I know there are probably better ways to deal with scoping variables, but for 
    # right now, this is what I have
    if var1:
        logging.info("Already set var1")
    else:
        var1, var2 = foo()   

    logger.info("Got event {}".format(event))
    return {"response": "Success!"}

I've built and locally tested docker containers for Lambda before on OSX and it works perfectly. However, moving over to a Windows machine, my Dockerfile (and subsequent containers) have started behaving strangely.

My Dockerfile:

ARG FUNCTION_DIR="/function"
ARG REQS_DIR="/requirements"
ARG TEST_DIR="tests"

FROM python:3.9-buster AS build-image
RUN apt-get update && apt-get install -y g++ make cmake unzip libcurl4-openssl-dev
ARG FUNCTION_DIR
ARG REQS_DIR
RUN mkdir -p ${REQS_DIR}
RUN mkdir -p ${FUNCTION_DIR}
COPY requirements.txt ${REQS_DIR}
RUN python3 -m pip install pip --upgrade
RUN python3 -m pip install -r ${REQS_DIR}/requirements.txt --target ${FUNCTION_DIR} 
RUN python3 -m pip install --target ${FUNCTION_DIR} awslambdaric 
ADD ./src ${FUNCTION_DIR}/src

FROM public.ecr.aws/lambda/python:3.9 AS local-test
ARG FUNCTION_DIR
COPY --from=build-image ${FUNCTION_DIR} ${LAMBDA_TASK_ROOT}
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/local/bin/aws-lambda-rie
RUN chmod +x /usr/local/bin/aws-lambda-rie
ENTRYPOINT [ "/usr/local/bin/aws-lambda-rie", "python3", "-m", "awslambdaric" ]
CMD ["src.main.handler"]

To build, I am running docker buildx build --no-cache -t local-test:latest --target local-test -f ./path/to/my/Dockerfile .

My docker-compose looks like:

version: '3.3'
services:
  local-test:
    ports:
      - '9000:8080'
    image: 'local-test:latest'
    container_name: 'local-tester'
    env_file:
      - '.env'

After running docker-compose up --detach, I run: curl -X POST "http://localhost:9000/2015-03-31/functions/function/invocations" -d "{}"

This request hangs seemingly indefinitely with no change to the logs within the container. Instead, the logs just say "exec 'python' (cwd=/var/task, handler=src.main.lambda_handler)" and nothing else.

I have moved everything into ENTRYPOINT, which did nothing. I even changed the command to be

CMD ["src.main.thing"]
# This should cause it to break

The above did not cause the container to break, and it started up just fine, when it shouldn't.

I moved everything to CMD (getting rid of ENTRYPOINT entirely) and got the error "entrypoint requires the handler name to be the first argument", which was odd.

The same is true if I remove both CMD and ENTRYPOINT, with it returning the error "entrypoint requires the handler name to be the first argument".

Exec-ing into the container shows that AWS_LAMBDA_RUNTIME_API is not set, which is odd. This causes the awslambdaric library to complain and error out if I run python3 -m awslambdaric src.main.lambda_handler

However, if I run in that shell export AWS_LAMBDA_RUNTIME_API=3.9, and then run python3 -m awslambdaric src.main.lambda_handler, the handler code actually runs! I can't run the full runtime emulator command, since it complains of ports being in-use. I only have this one container running.

Running docker with wsl. WSL is updated and I'm using Ubuntu 20.04.3 as my default.

Genuinely stuck, as I've been reading through all of the other related issues to this and attempted those fixes, to no avail.

Answer 1

Figured it out! I noticed that, despite removing my images and pruning docker via docker image prune and then docker rmi $(docker ps -aq), my images were still referencing layers that "already existed". Furthermore, my entrypoints were executing even when I commented them out.

To fix this issue, the following needs to be done on Windows using WSL (can't confirm with HyperV backend):

1. Open the Docker Desktop app
2. Hit the debug icon (the little bug in the upper left).
3. Hit Purge data".
4. Purge data for WSL only.

Why in the world this has to be done this way is beyond me, but this will fix any caching issues and should fix your Docker containers not behaving as expected.