Unable to import psycopg2 from AWS Lambda Layer due to problem with _psycopg module

Question

I've been trying for several days to create an AWS Lambda function that uses psycopg2. After looking at numerous tutorials and lots of trial and error, I have the following process.

I build and run a docker container from the following Dockerfile:

FROM public.ecr.aws/lambda/python:3.12

# Setup
COPY requirements.txt .
RUN pip install --upgrade pip
RUN mkdir python
RUN dnf install -y zip
RUN dnf clean all

# Install prereqs for psycopg2. Info: https://www.psycopg.org/docs/install.html#install-from-source
RUN dnf -y install gcc
RUN dnf -y install python3-devel
RUN dnf -y install libpq-devel

# Install psycopg2 into newly created directory named 'python'
RUN pip install -r requirements.txt -t python

# Zip directory
RUN zip -r9 python.zip python

# Needed or it complains about entry point
ENTRYPOINT ["bash"]

Then I copy python.zip out of the container and upload it to Lambda as a layer. The content of requirements.txt is simply one line with the text psycopg2.

When I try to import psycopg2 in my Lambda function, I get this result:

{
  "errorMessage": "Unable to import module 'main': No module named 'psycopg2._psycopg'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "0e68c374-83b9-4531-8ef7-26bf84cc3270",
  "stackTrace": []
}

I have unzipped python.zip and found a file called _psycopg.cpython-312-aarch64-linux-gnu.so. If I rename this file to _psycopg.so and reupload as a new layer, I get:

{
  "errorMessage": "Unable to import module 'main': /opt/python/psycopg2/_psycopg.so: cannot open shared object file: No such file or directory",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "e47dc97c-733d-4ecf-bf47-acf5d24be0a0",
  "stackTrace": []
}

It seems like AWS Lambda can't deal with this .so file for whatever reason, even though it works on the Docker container. I have seen some other StackOverflow questions addressing a similar issue but none of them have been very helpful. Many answers suggest using libraries on GitHub like awslambda-psycopg2 that are years old and don't support the latest python version. I would like to avoid this if possible. I'm very frustrated at this point and would really appreciate any help. Thanks in advance!

Disclaimer: I'm new to AWS and Docker so if it's obvious that I have no clue what I'm doing, that's why.

EDIT: I'm on macOS Sonoma 14.3 with M2 chip