Right way to use docker rootless mode on ZFS filesystem

Question

I configure my docker to be used with rootless mode and it works fine. But I changed my data-dir to store all the docker's data in a zfs filesystem. It appears that it is not working directly and I am looking for the right way to setup my system.

The storage driver is well configured and I obtain the following error

docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Extracting [==================================================>]  2.479kB/2.479kB
docker: failed to register layer: exit status 1: "/sbin/zfs fs create -o mountpoint=legacy xxxx/main/3a2069b67391755d42342cbfa9c0a78cb779759787e219bd8a80c4c2d51c2e96" => cannot create 'xxxx/main/3a2069b67391755d42342cbfa9c0a78cb779759787e219bd8a80c4c2d51c2e96': permission denied

I assumed that this error is completely normal but how can we force securely to allow to create mountpoint without root rights.

Answer 1

As of ZFS 2.2.0 (Oct. 13, 2023), overlayfs is supported on top of a zfs filesystem:

Linux container support (#12209, #14070, #14097, #12263) - Added support for Linux-specific container interfaces such as renameat(2), support for overlayfs, idmapped mounts in a user namespace, and namespace delegation support for containers.

This means that you can have /var/lib/docker (or it's rootless equivalent) on a ZFS filesystem, as long as you select the overlay2 storage driver.

Systems which provide their own ZFS builds such as TrueNAS (Core/Scale), Proxmox Virtual Environment (Proxmox VE) and NixOS may not have release 2.2.0 available immediately - at the time of writing (Oct. 25, 2023), Proxmox VE (v8.0.2-7) is still on zfs-2.1.13-pve1 and zfs-kmod-2.1.12-pve1.

Answer 2

The zfs storage driver is unfortunately not supported in rootless

From https://docs.docker.com/engine/security/rootless/

Known limitations

Only the following storage drivers are supported:

• overlay2 (only if running with kernel 5.11 or later, or Ubuntu-flavored kernel)
• fuse-overlayfs (only if running with kernel 4.18 or later, and fuse-overlayfs is installed)
• btrfs (only if running with kernel 4.18 or later, or ~/.local/share/docker is mounted with user_subvol_rm_allowed mount option)
• vfs