How to cache package manager downloads for docker builds?

Question

If I run composer install from my host, I hit my local composer cache:

  - Installing deft/iso3166-utility (1.0.0)
    Loading from cache

Yet when building a container having in its Dockerfile:

RUN composer install -n -o --no-dev

I download all the things, e.g.:

  - Installing deft/iso3166-utility (1.0.0)
    Downloading: 100%         

It's expected, yet I like to avoid it. As even on a rebuilt, it would also download everything again.

I would like to have a universal cache for composer that I could also reshare for other docker projects.

I looked into this and found the approach to define a volume in the Dockerfile:

ENV COMPOSER_HOME=/var/composer
VOLUME /var/composer

I added that to my Dockerfile, and expected to only download the files once, and hit the cache afterwards.

Yet when I modify my composer, e.g. remove the -o flag, and rerun docker build ., I expected to hit the cache on build, yet I still download the vendors again.

How are volumes supposed to work to have a data cache inside a docker container?

Answer 1

Anyone looking for this from now should be able to use

RUN --mount=type=cache,target=/root/.composer/ composer install

This general principle seems to work for most package installers, you just need to find the right cache dir (npm uses $HOME/.npm, etc.), and it will obviously change if you are not building as root.

Answer 2

I would like to have a universal cache for composer that I could also reshare for other docker projects.

Using a shared volume for the Composer cache works great when working with containers. If you want to go broader than just containers, and use a shared cache for e.g. local development as well, I've developed a solution for that called Velocita - how it works.

Basically, you use one global Composer plugin for local projects and inside and build containers. This not only speeds up downloads tremendously, it also helps with 3rd party outage for example.

Answer 3

I would consider utilizing the $HOME/.composer/cache/files directory. This is where composer reads/write to when using composer install.

If you are able to mount it from your host to your container that would work. Also you could just tar it up after each time your run composer install and then drop that in before you run composer install the next time.

This is loosely how Travis CI recommends doing this.

Also, consider using the --prefer-dist flag with your composer install command.

Info on that can be found here: https://getcomposer.org/doc/03-cli.md#install

--prefer-dist: Reverse of --prefer-source, composer will install from dist if possible. This can speed up installs substantially on build servers and other use cases where you typically do not run updates of the vendors. It is also a way to circumvent problems with git if you do not have a proper setup.

Some references on utilizing the composer cache for you:

• https://blog.wyrihaximus.net/2015/07/composer-cache-on-travis/
• https://github.com/travis-ci/travis-ci/issues/4579

Answer 4

I found two ways of dealing with this problem, yet none deal with composer volumes anymore.

1. Fasten composer download process: Use hirak/prestissimo

   composer global require "hirak/prestissimo:^0.3"
   

With Composer 2.0, the above step is no longer required for faster downloads. In fact, it won't install on Composer 2.0 environments.

2. Force docker to use a cached composer install.
   Docker uses a cache on a RUN if the added files didn't change. If you only do COPY . /your-php-app, docker build will refresh all the cashes and re-run composer install even if only one unrelated file in the source tree changed.
   In order to make docker build to run composer install only install on package changes, one has to add composer.json and composer.lock file before adding the source files. Since one also needs the source files anyway, one has to use different folders for composer install and rsync the content back to the then added folder; furthermore one then has to run the post-install scripts manually.
   It should look something like this (untested):

   WORKDIR /tmp/
   COPY composer.json composer.lock ./
   RUN composer install -n -o --no-dev --no-scripts
   
   WORKDIR /your-php-app/ 
   COPY . /your-php-app/
   RUN rsync -ah /tmp/* /your/php-app/
   RUN composer run-script post-install-cmd
   

or combine the two =)

Answer 5

Use the experimental feature : Docker buildkit (Supported Since docker 18.09, docker-compose 1.25.4)

In your dockerfile

# syntax=docker/dockerfile:experimental
FROM ....
# ......  
RUN --mount=type=cache,target=/var/composer composer install -n -o --no-dev

Now before building, make sure the env var is exported:

export DOCKER_BUILDKIT=1
docker build ....

If you are using docker-compose, make sure to export also COMPOSE_DOCKER_CLI_BUILD :

export COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1
docker-compose build ...

If it does not work with docker-compose, make sure your docker-compose version is above 1.25.4

docker-compose version