Goal
I'm running docker in a live CD and I want to cache the results of docker build, mostly, for when I reboot into this live CD. My idea was to set up an overlayfs in /var/lib/docker. So, I have the below in /etc/fstab:
overlay /var/lib/docker overlay lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work,x-systemd.requires=/mnt/root,x-systemd.requires=/var/lib/docker,nofail 0 2
However, I also want to use the overlay2 storage driver. I'm having a hard time getting both to cooperate. There are 2 ways that I've tried to get this to work.
Approach 1 (docker service before mount)
1a. Start docker.{socket,service} systemd units.
sudo systemctl restart docker.service # restarts docker.socket automatically
1b. Start overlayfs
sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work
Both of these steps execute successfully. However, if I try to build/run a new image then I get the following error.
Failure Kind 1
$ make up
mkdir -p .dynamodb-data/
docker-compose up -d --build
Pulling dynamodb (amazon/dynamodb-local:)...
latest: Pulling from amazon/dynamodb-local
2cbe74538cb5: Pull complete
2b0e6045b331: Extracting [==================================================>] 150MB/150MB
667cfda26bc2: Download complete
ERROR: failed to register layer: Error processing tar file(exit status 1): failed to mknod("/etc/sysconfig/clock", S_IFCHR, 0): operation not permitted
make: *** [Makefile:15: up] Error 1
Failure Kind 2
$ docker run -it --rm alpine
docker: Error response from daemon: open /var/lib/docker/overlay2/011d016883746bd72b20f298ecd73545a982019acd9eb3f600ff3b2bd4768888/committed: no such file or directory.
See 'docker run --help'.
Approach 2 (mount before docker service)
2a. Start overlayfs
sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work
2b. Start docker.{socket,service} systemd units.
sudo systemctl restart docker.service # restarts docker.socket automatically
2a. succeeds (of course), but 2b. fails. I've tried to capture some of the systemd logs, below, to try to help diagnose this problem.
Failure Kind 1
Running Command
$ sudo systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
journalctl
$ journalctl -u docker
-- Journal begins at Sat 2022-01-08 12:36:55 EET, ends at Sat 2022-01-08 13:30:15 EET. --
Jan 08 12:37:11 johnos systemd[1]: Starting Docker Application Container Engine...
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.548295176+02:00" level=info msg="Starting up"
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575509563+02:00" level=info msg="libcontainerd: started new containerd process" pid=2346
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575558875+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575566468+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575590744+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575605990+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.212959027+02:00" level=info msg="starting containerd" revision=v1.4.11 version=v1.4.11
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229766943+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229835283+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233082343+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /r>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233119579+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233257178+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (tmpfs) must be a btrfs filesystem to be>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233270945+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233289127+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233297875+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233320748+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233391289+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233485569+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233500101+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233516933+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233524181+02:00" level=info msg="metadata content store policy set" policy=shared
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233604310+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233619120+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233663958+02:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233690208+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233702617+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233713672+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233728996+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233740040+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233751585+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233763065+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233773387+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233828995+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233872779+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234064302+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234082499+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234118718+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234131212+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234141269+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234150671+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234160162+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234172118+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234181945+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234191381+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234200863+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234238659+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234251883+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234262989+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234272388+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234387869+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234413901+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234435704+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234447184+02:00" level=info msg="containerd successfully booted in 0.021879s"
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237690927+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237706112+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237717947+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237724235+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238078323+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238095004+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238105606+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238112459+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.293257377+02:00" level=info msg="Loading containers: start."
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.360740919+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 08 12:37:14 johnos dockerd[2267]: time="2022-01-08T12:37:14.902691695+02:00" level=info msg="Loading containers: done."
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113537187+02:00" level=info msg="Docker daemon" commit=v20.10.12 graphdriver(s)=overlay2 version=20.10.12
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113579097+02:00" level=info msg="Daemon has completed initialization"
Jan 08 12:37:15 johnos systemd[1]: Started Docker Application Container Engine.
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.121263835+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.124035385+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:42:57 johnos systemd[1]: Stopping Docker Application Container Engine...
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199583930+02:00" level=info msg="Processing signal 'terminated'"
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199951358+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.200178572+02:00" level=info msg="Daemon shutdown complete"
Overlay on top of overlay is not supported by the kernel. There is no docker configuration that will change that. You can find a list of supported backing filesystems in Docker's documentation. The least bad option if your backing filesystem is overlay is going to be fuse-overlayfs, which runs in user space. And the easiest but also worst option will be vfs, aka the native storage driver, which is a full copy of the filesystem for each layer and container (no overlay at all).