Unable to import module 'lambda function': libldap_r-2.4.so.2 cannot open shared objectfile: No such file or directory

Question

I've been dealing with this issue for the past few days now. I am trying to create a lambda layer with the following modules: cryptography, ldap3, and bonsai to use in my lambda_function that has a runtime of python3.11 and architecture type of x86_64. I have been using AWS CloudShell to create the lambda_layer zip file.

This is how I installed cryptography and ldap3:

-pip3.11 install \    
    --platform manylinux2014_x86_64 \
    --target=my-lambda-function \
    --implementation cp \
    --python-version 3.11.8 \
    --only-binary=:all: --upgrade \
    cryptography

-pip3.11 install \    
    --platform manylinux2014_x86_64 \
    --target=my-lambda-function \
    --implementation cp \
    --python-version 3.11.8 \
    --only-binary=:all: --upgrade \
    ldap3

These two modules work just fine I can call the module in my lambda function, the issue is with the bonsai module. Unfortunately, I can't install the bonsai module like this:

pip3.11 install \    
    --platform manylinux2014_x86_64 \
    --target=my-lambda-function \
    --implementation cp \
    --python-version 3.11.8 \
    --only-binary=:all: --upgrade \
    bonsai

Because I get the following error when I run this command:

ERROR: Could not find a version that satisfies the requirement bonsai (from versions:none)
ERROR: No matching distribution found for bonsai

Therefore this is how I am installing bonsai:

pip3.11 install bonsai --target=my-lambda-function

But when I try to call the bonsai module in my lambda function I get this error:

Unable to import module 'lambda function': libldap_r-2.4.so.2 cannot open shared objectfile: No such file or directory

Does anybody know how to resolve this issue, I've completely hit a brick wall. Any help is welcome!

Answer 1

It's much easier to build a lambda layer by using the Serverless Framework. I've built the layer for you. You can download it through this google drive link. This layer is for the x86_64 python3.11 runtime. Remember that you need to set the environment variable LD_LIBRARY_PATH to /opt/lib before using this layer.

Test code:

import json
import cryptography
import ldap3
import bonsai

def lambda_handler(event, context):
    # TODO implement
    print(cryptography.__version__)
    print(ldap3.__version__)
    print(bonsai.__version__)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Test result:

Test Event Name
test

Response
{
  "statusCode": 200,
  "body": "\"Hello from Lambda!\""
}

Function Logs
START RequestId: 3cfbef06-dd11-4fba-bec8-48004ad2e959 Version: $LATEST
42.0.5
2.9.1
1.5.2
END RequestId: 3cfbef06-dd11-4fba-bec8-48004ad2e959
REPORT RequestId: 3cfbef06-dd11-4fba-bec8-48004ad2e959  Duration: 8.01 ms   Billed Duration: 9 ms   Memory Size: 128 MB Max Memory Used: 67 MB  Init Duration: 506.47 ms

Request ID
3cfbef06-dd11-4fba-bec8-48004ad2e959

If you don't trust my code, you can follow the steps below to build it yourself.

Step 1: Add a requirements.txt

cryptography==42.0.5
ldap3==2.9.1
bonsai==1.5.2

Step 2: Add a Dockerfile

FROM public.ecr.aws/sam/build-python3.11:latest-x86_64
RUN yum -y install openldap-devel

Step 3: Modify serverless.yml

service: example

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerFile: Dockerfile
    dockerizePip: true
    layer: true
    dockerExtraFiles:
      - /lib64/libldap-2.4.so.2
      - /lib64/liblber-2.4.so.2
      - /lib64/libsasl2.so.3
      - /lib64/libssl3.so
      - /lib64/libsmime3.so
      - /lib64/libnss3.so
      - /lib64/libnssutil3.so

provider:
  name: aws
  runtime: python3.11
  stage: dev
  region: us-east-1
  profile: dummy

functions:
  dummy:
    handler: dummy.handler

Step 4: Build the layer

Simply do sls package -p pkg, then the layer zip named pythonRequirements.zip will be in pkg directory.

Step 5: Move .so files to a separate directory

Due to a bug in serverless-python-requirements, you need to unzip the pythonRequirements.zip, create a lib directory, move .so files into the new dir, and re-zip it.

Answer 2

You could try installing from the repo; pip install https://github.com/BonsaiAI/bonsai-python

You could build in CodeBuild and push the artifact to s3. This article talks about using CloudFormation to create the layer with a ref to an s3 artifact; https://www.linkedin.com/pulse/lambda-layer-creation-cloudformation-martin-macecek/

You could try to build inside a docker image with the same os as the lambda target os.

That .so file isn't available to the lambda os. I think that's the problem to work around somehow.