Can Selenium and Chrome buildpack installs be cached on HerokuCI?

Question

I've just started using HerokuCI. I have a few Selenium tests so I've added buildpacks in my app.json.

{
  "buildpacks": [
    { "url": "heroku/ruby" }
  ],
  "environments": {
    "test": {
      "addons": ["heroku-postgresql:in-dyno", "heroku-redis:in-dyno"],
      "buildpacks": [
        { "url": "heroku/ruby" },
        { "url": "https://github.com/notvad/heroku-buildpack-selenium" },
        { "url": "https://github.com/heroku/heroku-buildpack-chromedriver" },
        { "url": "https://github.com/heroku/heroku-buildpack-google-chrome" }
      ],
      "env": {
        "REDIS_PROVIDER": "REDIS_URL",
        "RAILS_ENV": "test",
        "RACK_ENV": "test",
        "MALLOC_ARENA_MAX": 2,
        "LANG": "en_US.UTF-8",
        "TEST_API": 1
      },
      "scripts": {
        "test-setup": "./bin/setup-cc-test-reporter",
        "test": "./bin/test-with-cc-test-reporter"
      }
    }
  }
}

These keep getting rebuilt along with all their dependencies.

-----> Selenium app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
-----> Updating apt caches
       Hit:1 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for xserver-xorg-core libgl1-mesa-dri xvfb libglide3 xfs unzip
       Reading package lists...
       Building dependency tree...
       Package xfs is not available, but is referred to by another package.
       This may mean that the package is missing, has been obsoleted, or
       is only available from another source

E: Package 'xfs' has no installation candidate
-----> Installing adwaita-icon-theme_3.28.0-1ubuntu1_all.deb
-----> Installing at-spi2-core_2.28.0-1_amd64.deb

...and so on...

-----> Downloading and installing Selenium
-----> Writing profile script
-----> chromedriver app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
-----> Looking up latest chromedriver version...
-----> Downloading chromedriver v2.43...
Archive:  /tmp/chromedriver.zip
  inflating: /app/.chromedriver/bin/chromedriver  
-----> Creating chromedriver export scripts...
-----> Google Chrome app detected
bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory
bash: /tmp/buildpacks/b7af5642714be4eddaa5f35e2b4c36176b839b4abcd9bfe57ee71c358d71152b4fd2cf925c5b6e6816adee359c4f0f966b663a7f8649b0729509d510091abc07/export: No such file or directory
bash: /tmp/buildpacks/f55d0db8af2b8c70f2c0e028ef5430955852d9e4f07a06e333e54eb5264a81b9d1e77d1cc02851b4c7da30f6da47b13959e18a24c630212b899920d05f32a566/export: No such file or directory
bash: /tmp/buildpacks/27353aa3b5edb2f8361186206e863f5d481f02805d02f0efdec439f3d30349a4fbdad2951af33a0492023fb0d90d26dafda6fe76f1b2d34f68564a98545f7022/export: No such file or directory
-----> Installing Google Chrome from the stable channel.
-----> Updating apt caches
       Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
       Hit:2 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
       Hit:3 http://archive.ubuntu.com/ubuntu bionic-security InRelease
       Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
       Reading package lists...
-----> Fetching .debs for gconf-service

...and so on...

This is taking up most of the test run time.

Can these installs be cached?

Answer 1

After various search and try, I bring good news.

The key issue is your errors in log files, like:

bash: /tmp/buildpacks/b5ae60a477f2ba582d00a4ba23348059d40dea69cea6d982947f7c6afa72d2b8d69252449d43cbcf97a23341fd57300a3fef3224ace95482e1c0deaf8a844e4d/export: No such file or directory

This auto-generated directory is obviously a cache directory one, which is generated automatically when not fixed.

Fore instance, this can be confirmed inside the heroku-buildpack-selenium buildpack, defined in your app.json file:

# parse and derive params
BUILD_DIR=$1
CACHE_DIR=$2
LP_DIR=`cd $(dirname $0); cd ..; pwd`

It is confirmed by this official documentation I found:

The contents of CACHE_DIR will be persisted between builds. You can cache the results of long processes like dependency resolution here to speed up future builds.

So the key solution is to define this CACHE_DIR variable within your installation; so I guess it is either directly in your app.json file, or directly in your GitHub Hook.

Let me know if you need further help.

Answer 2

If you have the caches enabled as described here it does not seem like there is much more you can do to cache the installs. If this is a serious problem you could always point to local packages that you cloned from the remote repos to speed things up.