Psycopg3 - no pq wrapper available

Question

I am facing this error when I try to run psycopg3 on AWS Lambda.

[ERROR] Runtime.ImportModuleError: Unable to import module 'function': no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: libpq library not found

I reference it as a dependency in my .venv based on this in my setup.py

    install_requires=[
        'boto3',
        'pydantic',
        'python-dotenv',
        'psycopg',
        'twine',
        'wheel',
        'xlsxwriter'
    ],
    extras_require={
        'binary': ['psycopg-binary'],
    },

I have copied the installation process as the official site.

Answer 1

As per my understanding, you will need a binary too.

Some External libraries like GPG, psycopg3 require them to be statically compiled as a binary using Amazon Linux 2 OS to run on amazon lambda. You can build it using a docker image and zip it and push as lambda layer to be used in lambda. This process is not straightforward.

Instead I would suggest to take a look at pg8000 driver. https://github.com/tlocke/pg8000

references to use: https://stackoverflow.com/a/72571215/5235168

Answer 2

So, the solution for me was to remove the psycopg[binary] dependency from the main pypi repository package, and only reference it when deploying the lambda, which needs to be done via docker (I suppose once deployment is hooked up to an AWS VM it will be better). The pypi package will only have psycopg as a dependency in requirements.txt and also needs docker deployment.

Here is the code for final deployment of the function:

mkdir ./lambda-code

cp -r function.py exception.py requirements.txt ./lambda-code

MSYS2_ARG_CONV_EXCL="*" docker run -u $(id -u):$(id -g) \
    -v "$(pwd)/lambda-code:/lambda-code" \
    -v "$(pwd)/requirements.txt:/requirements.txt" \
    -v "$(pwd)/function.py:/lambda-code/function.py" \
    -v "$(pwd)/exception.py:/lambda-code/exception.py" \
    -v "$HOME/.cache/pip:/.cache/pip" \
    python:3.10 sh -c "pip install -r /requirements.txt -t /lambda-code && \
                      pip install \
                      --platform manylinux2014_x86_64 \
                      --target=/lambda-code \
                      --implementation cp \
                      --python 3.10 \
                      --only-binary=:all: --upgrade \
                      'psycopg[binary]'"

cd lambda-code || exit
zip -r ../lambda_function.zip .

And then you may upload this zipped file to AWS lambda.