How to Have NaCL at AWS Lambda Properly Working?

Question

I am using Pychromeless repo with success at AWS lambda.

Now I need to use NaCL dependency, to decrypt a string, but I am getting

Unable to import module 'lambda_function': /var/task/lib/nacl/_sodium.abi3.so

with a complement of

invalid ELF header  

when running the function on AWS Lambda.

I know it is a problem specific related to AWS Lambda environment, because I can run the function on my Mac inside docker.

Here's my requirements.txt file

boto3==1.6.18
botocore==1.9.18
selenium==2.53.6
chromedriver-install==1.0.3
beautifulsoup4==4.6.1
certifi==2018.11.29
chardet==3.0.4
editdistance==0.5.3
future==0.17.1
idna==2.7
python-telegram-bot==10.1.0
requests==2.19.1
soupsieve==1.7.3
urllib3==1.23
PyNaCl==1.3.0

Here is the dockerfile

FROM lambci/lambda:python3.6
MAINTAINER [email protected]

USER root

ENV APP_DIR /var/task

WORKDIR $APP_DIR

COPY requirements.txt .
COPY bin ./bin
COPY lib ./lib

RUN mkdir -p $APP_DIR/lib
RUN pip3 install -r requirements.txt -t /var/task/lib

And the makefile:

clean:
    rm -rf build build.zip
    rm -rf __pycache__

fetch-dependencies:
    mkdir -p bin/

    # Get chromedriver
    curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
    unzip chromedriver.zip -d bin/

    # Get Headless-chrome
    curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip
    unzip headless-chromium.zip -d bin/

    # Clean
    rm headless-chromium.zip chromedriver.zip

docker-build:
    docker-compose build

docker-run:
    docker-compose run lambda src/lambda_function.lambda_handler


build-lambda-package: clean fetch-dependencies
    mkdir build
    cp -r src build/.
    cp -r bin build/.
    cp -r lib build/.
    pip install -r requirements.txt -t build/lib/.
    cd build; zip -9qr build.zip .
    cp build/build.zip .
    rm -rf build

Without the decryption part, the code works great. So the issue is 100% related to PyNaCl.

Any help on solving this?

Answer 1

I think you may try to setup PyNaCl like so:

SODIUM_INSTALL=system pip3 install pynacl

that will force PyNaCl to use the version of libsodium provided by AWS

see this

in the last version of PyNaClis updated to libsodium 1.0.16. so maybe it is not compatible with AWS

so you may remove PyNaCl from requirements.txt and add this to your Dockerfile:

RUN SODIUM_INSTALL=system pip3 install pynacl -t /var/task/lib

or maybe setup the dockerfile like this and keep PyNaCl in requirements.txt:

ARG SODIUM_INSTALL=system

Try also to setup sodium before installing PyNaCI:

RUN wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.15.tar.gz \ && tar xzf libsodium-1.0.15.tar.gz \ && cd libsodium-1.0.15 \ && ./configure \ && make install

Answer 2

Ok, this is how I did it. I had to build everything on an EC2 AMI Linux 2 instance.

amzn2-ami-hvm-2.0.20190823.1-x86_64-gp2 (ami-0a1f49a762473adbd)

After launching the instance, I use this script to install Python 3.6 (and pip) and to create and activate a virtual environment.

For the docker part, I followed this tutorial, not without some troubles on the way (had to

sudo yum install polkit

and

sudo usermod -a -G docker ec2-user

and

reboot

and

sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose

and

sudo chmod +x /usr/local/bin/docker-compose).

But anyway, I manage to work with docker on the EC2 instance, building the zip file and uploading it to Lambda environment, where everything worked fine, as I expected.

I thought docker was an environment independent from the host, but I guess that is not the case.