Importing the numpy C-extensions failed to run in local container (sam local invoke) whereas it runs perfectly locally ( python .\test.py)

Question

This question was asked many times before I know. After days of struggle, I have narrowed the issue to this: The code runs perfectly locally by using python .\test.py, but fails while ruining in the sam local invoke container. AWS supports python 3.9 so I tried to figure out the dependency combination for those imports:

import numpy as np
import pandas as pd
from scipy.signal import savgol_filter
from sktime.transformations.series.outlier_detection import HampelFilter
from sklearn.linear_model import LinearRegression
from scipy.signal import find_peaks, peak_prominences

The code runs locally without problems but fails to run in the container. Please help

I figured out that sktime will support python 3.9 but depends on numpy>=1.21.0. On the other hand no matter what I tried.. I couldn't figure out what dependencies will work on AWS python 3.9. I have tries dto install from wheels and by trying different versions of numpy, but it is always the same error.

I ended up limiting the requirements like so:

numpy==1.21.0
pandas
sktime

This installed what it needed. Then I figured out that I can run it locally. No issues. So the problem must be with the container sam local invoke runs.

Note that I can use AWS layer for pandas 3.9 but I also need sktime and that didn't work for me (not to mention the 250M upload limit that I leave for later straggle).

Answer 1

Ok.. I have a better understanding of the whole issue. Here is what I have learned:

1. When you install Python for AWS you should install the supported version (3.9 as for now). This will make all requirements.txt installations match Python 3.9.
2. So.. all installations should match without any issues.
3. Still running locally did not work for me. I do not know why!!!!
4. One solution involves attaching the lambda to EFS that includes many steps and manual work. You don't want to go there.
5. The solution for me was using Lambda Images. This was easier then I have expected. The steps includes some docker configurations and having the yaml use the docker file, and then 'sam deploy' will publish a Docker image for you to Elastic Container Register - ECR. See: Creating Lambda container images Using container image support for AWS Lambda with AWS SAM

The first link will lead to some other supporting link within AWS documentation.

Hope it will help someone.