Heroku Python binary library dependency

Question

Heroku Python binary library dependency

2.5k Views Asked by Joseph Victor Zammit At 19 October 2024 at 21:24

I need to run code on my Heroku Django project that requires the Python package dm.xmlsec.binding. This in turn, depends on having the binary apt package libxmlsec1 installed, usually via apt-get install.

That is how I have things running locally on an Ubuntu vagrant box.

Deploying to Heroku: Approach

I have two "build packs" in my application. Having multiple buildpacks is enabled by using heroku-buildpack-multi.

Since my application is a Python one, my .buildpacks contents are therefore:

https://github.com/ddollar/heroku-buildpack-apt
https://github.com/heroku/heroku-buildpack-python

The first buildpack, which refers to heroku-buildpack-apt, requires an Aptfile to specify apt packages; in our case the contents of this file are:

python-setuptools
libxmlsec1
libxmlsec1-dev
swig
python-m2crypto

The second buildpack is the "Heroku buildpack for Python apps, powered by pip".

So far so good.

Deploying to Heroku: Problem

The problem is that, while apt packages look like being installed successfully, when pip install actually runs, it seems not to find the apt packages installed.

If apt packages are being installed correctly, why is Python not finding them in the subsequent pip install?

This is the full git push heroku log:

git push heroku
Fetching repository, done.
Counting objects: 12, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (9/9), 932 bytes | 0 bytes/s, done.
Total 9 (delta 3), reused 0 (delta 0)

-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/ddollar/heroku-buildpack-apt
=====> Detected Framework: Apt
-----> Updating apt caches
       Ign http://archive.ubuntu.com trusty InRelease
       Ign http://archive.ubuntu.com trusty-security InRelease
       Ign http://archive.ubuntu.com trusty-updates InRelease
       Hit http://archive.ubuntu.com trusty Release.gpg
       Get:1 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
       Get:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
       Hit http://archive.ubuntu.com trusty Release
       Get:3 http://archive.ubuntu.com trusty-security Release [62.0 kB]
       Get:4 http://archive.ubuntu.com trusty-updates Release [62.0 kB]
       Hit http://archive.ubuntu.com trusty/main amd64 Packages
       Hit http://archive.ubuntu.com trusty/universe amd64 Packages
       Hit http://archive.ubuntu.com trusty/main Translation-en
       Hit http://archive.ubuntu.com trusty/universe Translation-en
       Get:5 http://archive.ubuntu.com trusty-security/main amd64 Packages [153 kB]
       Hit http://archive.ubuntu.com trusty-security/main Translation-en
       Get:6 http://archive.ubuntu.com trusty-updates/main amd64 Packages [356 kB]
       Hit http://archive.ubuntu.com trusty-updates/main Translation-en
       Fetched 635 kB in 2s (247 kB/s)
       Reading package lists...
-----> Fetching .debs for python-setuptools
       Reading package lists...
       Building dependency tree...
       The following extra packages will be installed:
         python-pkg-resources
       Suggested packages:
         python-distribute python-distribute-doc
       The following NEW packages will be installed:
         python-pkg-resources python-setuptools
       0 upgraded, 2 newly installed, 0 to remove and 38 not upgraded.
       Need to get 292 kB of archives.
       After this operation, 1017 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main python-pkg-resources all 3.3-1ubuntu1 [61.9 kB]
       Get:2 http://archive.ubuntu.com/ubuntu/ trusty/main python-setuptools all 3.3-1ubuntu1 [230 kB]
       Fetched 292 kB in 0s (408 kB/s)
       Download complete and in download only mode
-----> Fetching .debs for libxmlsec1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxmlsec1
       0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
       Need to get 0 B/115 kB of archives.
       After this operation, 436 kB of additional disk space will be used.
       Download complete and in download only mode
-----> Fetching .debs for libxmlsec1-dev
       Reading package lists...
       Building dependency tree...
       The following extra packages will be installed:
         libnspr4 libnspr4-dev libnss3 libnss3-dev libnss3-nssdb libxmlsec1
         libxmlsec1-gcrypt libxmlsec1-gnutls libxmlsec1-nss libxmlsec1-openssl
       The following NEW packages will be installed:
         libnspr4 libnspr4-dev libnss3 libnss3-dev libnss3-nssdb libxmlsec1
         libxmlsec1-dev libxmlsec1-gcrypt libxmlsec1-gnutls libxmlsec1-nss
         libxmlsec1-openssl
       0 upgraded, 11 newly installed, 0 to remove and 38 not upgraded.
       Need to get 0 B/2579 kB of archives.
       After this operation, 14.1 MB of additional disk space will be used.
       Download complete and in download only mode
-----> Fetching .debs for swig
       Reading package lists...
       Building dependency tree...
       The following extra packages will be installed:
         swig2.0
       Suggested packages:
         swig-doc swig-examples swig2.0-examples swig2.0-doc
       The following NEW packages will be installed:
         swig swig2.0
       0 upgraded, 2 newly installed, 0 to remove and 38 not upgraded.
       Need to get 0 B/881 kB of archives.
       After this operation, 4412 kB of additional disk space will be used.
       Download complete and in download only mode
-----> Fetching .debs for python-m2crypto
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         python-m2crypto
       0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
       Need to get 0 B/156 kB of archives.
       After this operation, 831 kB of additional disk space will be used.
       Download complete and in download only mode
-----> Installing libnspr4-dev_2%3a4.10.7-0ubuntu0.14.04.1_amd64.deb
-----> Installing libnspr4_2%3a4.10.7-0ubuntu0.14.04.1_amd64.deb
-----> Installing libnss3-dev_2%3a3.17.1-0ubuntu0.14.04.1_amd64.deb
-----> Installing libnss3-nssdb_2%3a3.17.1-0ubuntu0.14.04.1_all.deb
-----> Installing libnss3_2%3a3.17.1-0ubuntu0.14.04.1_amd64.deb
-----> Installing libxmlsec1-dev_1.2.18-2ubuntu1_amd64.deb
-----> Installing libxmlsec1-gcrypt_1.2.18-2ubuntu1_amd64.deb
-----> Installing libxmlsec1-gnutls_1.2.18-2ubuntu1_amd64.deb
-----> Installing libxmlsec1-nss_1.2.18-2ubuntu1_amd64.deb
-----> Installing libxmlsec1-openssl_1.2.18-2ubuntu1_amd64.deb
-----> Installing libxmlsec1_1.2.18-2ubuntu1_amd64.deb
-----> Installing python-m2crypto_0.21.1-3ubuntu5_amd64.deb
-----> Installing python-pkg-resources_3.3-1ubuntu1_all.deb
-----> Installing python-setuptools_3.3-1ubuntu1_all.deb
-----> Installing swig2.0_2.0.11-1ubuntu2_amd64.deb
-----> Installing swig_2.0.11-1ubuntu2_amd64.deb
-----> Writing profile script
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python
=====> Detected Framework: Python
-----> Installing dependencies with pip
       Downloading/unpacking dm.xmlsec.binding==1.3.1 (from -r requirements.txt (line 1))
         Running setup.py (path:/tmp/pip_build_u45022/dm.xmlsec.binding/setup.py) egg_info for package dm.xmlsec.binding
           Error: cannot get XMLSec1 pre-processor and compiler flags; do you have the `libxmlsec1` development package installed?
           Complete output from command python setup.py egg_info:
           Error: cannot get XMLSec1 pre-processor and compiler flags; do you have the `libxmlsec1` development package installed?

       ----------------------------------------
       Cleaning up...
       Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_u45022/dm.xmlsec.binding
       Storing debug log for failure in /app/.pip/pip.log

 !     Push rejected, failed to compile Multipack app

To [email protected]:xxxxxxxxxxx.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]'

Original Q&A

There are 3 best solutions below

**fiatjaf** · Answer 1

I don't know, but maybe the answer is this:

With the amazing help of @PadraicCunningham I managed to find the solution. The problem turned out to be that heroku-buildpack-apt installs things in a newly created folder /app/.apt/ which was not in the PYTHONPATH.

So I added the relevant folder to my PYTHONPATH on heroku as follows:

heroku config:add PYTHONPATH=/app/.apt/usr/lib/python2.7/dist-packages/

**Raghav Mundhra** · Answer 2

Whenever there is an OS dependency for any package in python, I prefer building Docker images on Heroku for hosting applications.

Create a Docker file in your root directory. A Docker file looks like this:

FROM gcr.io/google-appengine/python
RUN apt-get update
RUN apt install libxmlsec1
RUN pip install requirements.txt
CMD python main.py

Write all the apt commands you want in the docker file and save it with name, Dockerfile . Do not give any extension to the file name.

Create a file named, heroku.yml. Paste the below lines of code to it:

build:
  docker:
    web: Dockerfile

Use these commands to push your application to Heroku:

1. git add heroku.yml

2 git commit -m "Add heroku.yml"

3 heroku stack:set container

4 git push heroku master

**Joël Franusic** · Answer 3

For what it's worth, I was able to get xmlsec1 working on Heroku by doing the following two things:

Using a buildpack to install xmlsec1.
Using a .profile.d script to configure $PATH to point at the xmlsec1 binary.

Installing xmlsec1 using a buildpack

I used heroku-buildpack-multi with the strydercorp/heroku-buildpack-xmlsec buildpack to install xmlsec1 into my Heroku instance. Note that this buildpack is for Cedar14, if you are on standard Cedar, you will want to use evenco/heroku-buildpack-xmlsec

Here is what my .buildpack file looks like:

$ cat .buildpacks 
https://github.com/heroku/heroku-buildpack-python
https://github.com/strydercorp/heroku-buildpack-xmlsec

Configuring $PATH with a .profile.d script

Once the xmlsec1 binary was installed on my Heroku instance, I needed to configure the $PATH variable to point at the xmlsec1 binary. Here is the .profile.d script I used to do that:

$ cat .profile.d/path.sh 
# https://devcenter.heroku.com/articles/profiled

# add xmlsec1 binary and libraries to the appropriate paths
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/app/.heroku/xmlsec1-1.2/lib/"
PATH="$PATH:/app/.heroku/xmlsec1-1.2/bin/"

Heroku Python binary library dependency

Deploying to Heroku: Approach

Deploying to Heroku: Problem

There are 3 best solutions below

Related Questions in PYTHON

Related Questions in PYTHON-2.7

Related Questions in HEROKU

Related Questions in APT

Trending Questions

Popular # Hahtags

Popular Questions