Compiling libgdal for AWS AMI fails

Question

I try to get libgdal installed in an AWS AMI 2 to be used in ECR, together with Python bindings. The build runs within a Docker container which is provided by ECR and contains Python 3.9.

# Pull the base image with python 3.9 as a runtime for your Lambda
FROM public.ecr.aws/lambda/python:3.9

# install tools (gcc to compile C code during installation, mesa-libGL for OpenCV)
RUN yum install -y yum-plugin-remove-with-leaves tar file.x86_64 sudo make gcc git unzip mesa-libGL gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64

RUN python3.9 -m pip install numpy==1.22.4

WORKDIR /tmp
RUN curl https://download.osgeo.org/proj/proj-6.1.1.tar.gz -o proj-6.1.1.tar.gz && \
tar -xvf proj-6.1.1.tar.gz && \
cd proj-6.1.1 && \
ls -l && \
./configure --disable-dependency-tracking && \
make && \
sudo make install && \
cd .. && \
rm -rf proj-6.1.1 && \
rm -f proj-6.1.1.tar.gz

RUN yum install -y which
RUN ls -lr `which python`

RUN git clone --branch patch/3.2.2.1 https://github.com/OSGeo/gdal.git && \
cd gdal/gdal && \
./configure --with-proj=/usr/local --with-python && \
make && \
sudo make install && \
cd ../.. && \
rm -rf gdal


RUN which gdalinfo && \
gdalinfo --version && \
ls -l /usr/include/gdal

But when make install is executed, I get following error:

(cd port; make install)
make[1]: Entering directory `/tmp/gdal/gdal/port'
for f in cpl_atomic_ops.h cpl_config_extras.h cpl_config.h cpl_conv.h cpl_csv.h cpl_error.h cpl_hash_set.h cpl_http.h cpl_json.h cplkeywordparser.h cpl_list.h cpl_minixml.h cpl_minizip_ioapi.h cpl_minizip_unzip.h cpl_minizip_zip.h cpl_multiproc.h cpl_odbc.h cpl_port.h cpl_progress.h cpl_quad_tree.h cpl_spawn.h cpl_string.h cpl_time.h cpl_virtualmem.h cpl_vsi.h cpl_vsi_error.h cpl_vsi_virtual.h gdal_csv.h cpl_auto_close.h ; do /tmp/gdal/gdal/install-sh -c -m 0644 $f /usr/local/include ; done
make[1]: Leaving directory `/tmp/gdal/gdal/port'
(cd gcore; make install)
make[1]: Entering directory `/tmp/gdal/gdal/gcore'
for f in gdal_frmts.h gdalgeorefpamdataset.h gdal.h gdaljp2abstractdataset.h gdaljp2metadata.h gdal_mdreader.h gdal_pam.h gdal_priv.h gdal_proxy.h gdal_rat.h rawdataset.h gdal_version.h ; do /tmp/gdal/gdal/install-sh -c -m 0644 $f /usr/local/include ; done
make[1]: Leaving directory `/tmp/gdal/gdal/gcore'
(cd frmts; make install)
make[1]: Entering directory `/tmp/gdal/gdal/frmts'
make -C vrt install
make[2]: Entering directory `/tmp/gdal/gdal/frmts/vrt'
/tmp/gdal/gdal/install-sh -c -m 0644 vrtdataset.h /usr/local/include
/tmp/gdal/gdal/install-sh -c -m 0644 gdal_vrt.h /usr/local/include
make[2]: Leaving directory `/tmp/gdal/gdal/frmts/vrt'
make -C mem install
make[2]: Entering directory `/tmp/gdal/gdal/frmts/mem'
/tmp/gdal/gdal/install-sh -c -m 0644 memdataset.h /usr/local/include
make[2]: Leaving directory `/tmp/gdal/gdal/frmts/mem'
make -C raw install
make[2]: Entering directory `/tmp/gdal/gdal/frmts/raw'
make[2]: Nothing to be done for `install'.
make[2]: Leaving directory `/tmp/gdal/gdal/frmts/raw'
make[1]: Leaving directory `/tmp/gdal/gdal/frmts'
(cd alg; make install)
make[1]: Entering directory `/tmp/gdal/gdal/alg'
for f in gdal_alg.h gdal_alg_priv.h gdalgrid.h gdalgrid_priv.h gdalpansharpen.h gdal_simplesurf.h gdalwarper.h ; do /tmp/gdal/gdal/install-sh -c -m 0644 $f /usr/local/include ; done
make[1]: Leaving directory `/tmp/gdal/gdal/alg'
(cd ogr; make install)
make[1]: Entering directory `/tmp/gdal/gdal/ogr'
for f in ogr_core.h ogr_feature.h ogr_geometry.h ogr_p.h ogr_spatialref.h ogr_srs_api.h ogrsf_frmts/ogrsf_frmts.h ogr_featurestyle.h ogr_api.h ogr_geocoding.h ogr_swq.h ; \
    do /tmp/gdal/gdal/install-sh -c -m 0644 $f /usr/local/include ; \
done
make[1]: Leaving directory `/tmp/gdal/gdal/ogr'
(cd gnm; make install)
make[1]: Entering directory `/tmp/gdal/gdal/gnm'
for f in gnm.h gnm_api.h gnmgraph.h ; \
    do /tmp/gdal/gdal/install-sh -c -m 0644 $f /usr/local/include ; \
done
make[1]: Leaving directory `/tmp/gdal/gdal/gnm'
(cd apps; make install)
make[1]: Entering directory `/tmp/gdal/gdal/apps'
for f in gdalinfo gdal_translate gdaladdo gdalwarp nearblack gdalmanage gdalenhance gdaltransform gdaldem gdallocationinfo gdalsrsinfo gdalmdiminfo gdalmdimtranslate gdal_viewshed gdal_create gdal_contour gdaltindex gdal_rasterize gdal_grid ogrinfo ogr2ogr ogrtindex ogrlineref testepsg gdalbuildvrt gnmmanage gnmanalyse ; do /bin/sh /tmp/gdal/gdal/libtool --mode=install --silent /tmp/gdal/gdal/install-sh -c $f /usr/local/bin ; done
/tmp/gdal/gdal/install-sh -c -m 0644 gdal_utils.h /usr/local/include
/bin/sh /tmp/gdal/gdal/libtool --mode=install --silent /tmp/gdal/gdal/install-sh -c gdal-config-inst /usr/local/bin/gdal-config
make[1]: Leaving directory `/tmp/gdal/gdal/apps'
(cd swig; make install)
make[1]: Entering directory `/tmp/gdal/gdal/swig'
for dir in python ; do (cd $dir; make install) || exit; done
make[2]: Entering directory `/tmp/gdal/gdal/swig/python'
----------------------------------------------------------------------

The GDAL python package has been installed in lib/python2.7/site-packages
Please ensure to add lib/python2.7/site-packages to your PYTHONPATH

----------------------------------------------------------------------
env PYTHONPATH=lib/python2.7/site-packages${PYTHONPATH:+:$PYTHONPATH} \
    python setup.py install --single-version-externally-managed --record=record.txt --prefix=/usr/local
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'extras_require'
  warnings.warn(msg)
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --single-version-externally-managed not recognized
WARNING: numpy not available!  Array support will not be enabled
make[2]: Leaving directory `/tmp/gdal/gdal/swig/python'
make[2]: *** [install] Error 1
make[1]: Leaving directory `/tmp/gdal/gdal/swig'
make[1]: *** [install] Error 2
make: *** [install-actions] Error 2

Before I tried libgdal version 3.2.2, but I needed to switch to 3.2.2.1 which contains a fix for the installation. I use 3.2.2.1 because this is what I got installed on my developer machine. Maybe I can use a newer version of libgdal, I just want to extract GPS coordinates for pixels from a georeferenced TIFF.

Answer 1

For future reference, I used this image:

https://github.com/lambgeo/docker-lambda

Right now it's meant to create a package.zip that you will use to deploy the lambda with the zip file archive method. If you need to run this image as a container AWS lambda image you need to overwrite in a intermediate dockerfile image the ENTRYPOINT of ghcr.io/lambgeo/lambda-gdal:3.5-python3.9 (or ghcr.io/lambgeo/lambda-gdal:3.5-python3.8) using

ENTRYPOINT  ["/lambda-entrypoint.sh"]

---

EDIT: In my opinion the aws python docker images are a little outdated, so in the end I switched to the official image from osgeo/gdal (at this time it's osgeo/gdal:ubuntu-small-3.6.1, based on Ubuntu 22.04, with python 3.10).

To reduce build time I prepared this custom intermediate docker image. Here I also add aws lambda support and my python packages requirements list:

# inspired by https://dev.to/gaborschulz/running-python-311-on-aws-lambda-1i7p
FROM osgeo/gdal:ubuntu-small-3.6.1 as build-image

LABEL maintainer="alessandro trinca <[email protected]>"

# Include global arg in this stage of the build
ARG LAMBDA_TASK_ROOT="/var/task"
ARG PYTHONPATH="${LAMBDA_TASK_ROOT}:${PYTHONPATH}:/usr/local/lib/python3/dist-packages"

RUN mkdir -p ${LAMBDA_TASK_ROOT}

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y g++ make cmake unzip libcurl4-openssl-dev python3-pip

# install required packages
COPY requirements_pip.txt ${LAMBDA_TASK_ROOT}/
RUN python -m pip install --target ${LAMBDA_TASK_ROOT} --upgrade -r ${LAMBDA_TASK_ROOT}/requirements_pip.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}/
RUN python -m pip install --target ${LAMBDA_TASK_ROOT} -r ${LAMBDA_TASK_ROOT}/requirements.txt

FROM osgeo/gdal:ubuntu-small-3.6.1

# Include global arg in this stage of the build
ARG LAMBDA_TASK_ROOT="/var/task"
ARG PYTHONPATH="${LAMBDA_TASK_ROOT}:${PYTHONPATH}:/usr/local/lib/python3/dist-packages"
ARG RIE="https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie"

# Set working directory to function root directory
WORKDIR ${LAMBDA_TASK_ROOT}

# Copy in the built dependencies
COPY --from=build-image ${LAMBDA_TASK_ROOT} ${LAMBDA_TASK_ROOT}

RUN curl -Lo /usr/local/bin/aws-lambda-rie ${RIE}
RUN chmod +x /usr/local/bin/aws-lambda-rie

COPY ./scripts/lambda-entrypoint.sh /lambda-entrypoint.sh
RUN chmod +x /lambda-entrypoint.sh
RUN ls -l /lambda-entrypoint.sh

ENTRYPOINT  ["/lambda-entrypoint.sh"]

You also need to tag the intermediate docker image using a ECR-friendly naming convention (you don't need to push it immediately however):

docker tag image-name:latest <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/image-name:latest

More details in the blog page.

The lambda-entrypoint.sh script:

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
  exec /usr/local/bin/aws-lambda-rie /usr/bin/python -m awslambdaric $@
else
  exec /usr/bin/python -m awslambdaric $@
fi

The requirements_pip.txt

pip
wheel
setuptools

Last my "final" docker image based on the intermediate image I prepared before:

FROM <AWS_ACCOUNT_ID>.dkr.ecr.eu-west-1.amazonaws.com/image-name:latest


ARG LAMBDA_TASK_ROOT="/var/task"
ARG PYTHONPATH="${LAMBDA_TASK_ROOT}:${PYTHONPATH}:/usr/local/lib/python3/dist-packages"

COPY app.py ${LAMBDA_TASK_ROOT}

CMD [ "app.lambda_handler" ]