Mounting NFS volume in Google Container Engine with Container OS (COS)

Question

After migrating the image type from container-vm to cos for the nodes of a GKE cluster, it seems no longer possible to mount a NFS volume for a pod.

The problem seems to be missing NFS client libraries, as a mount command from command line fails on all COS versions I tried (cos-stable-58-9334-62-0, cos-beta-59-9460-20-0, cos-dev-60-9540-0-0).

sudo mount -t nfs mynfsserver:/myshare /mnt

fails with

mount: wrong fs type, bad option, bad superblock on mynfsserver:/myshare,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

But this contradicts the supported volume types listed here: https://cloud.google.com/container-engine/docs/node-image-migration#storage_driver_support

Mounting a NFS volume in a pod works in a pool with image-type container-vm but not with cos.

With cos I get following messages with kubectl describe pod:

MountVolume.SetUp failed for volume "kubernetes.io/nfs/b6e6cf44-41e7-11e7-8b00-42010a840079-nfs-mandant1" (spec.Name: "nfs-mandant1") pod "b6e6cf44-41e7-11e7-8b00-42010a840079" (UID: "b6e6cf44-41e7-11e7-8b00-42010a840079") with: mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1 nfs []
Output: Mount failed: Mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1]
Output: mount.nfs: Failed to resolve server singlefs-1-vm: Temporary failure in name resolution

Answer 1

Martin, are you setting up the mounts manually (executing mount yourself), or are you letting kubernetes do it on your behalf via a pod referencing an NFS volume?

The former will not work. The later will. As you've discovered COS does not ship with NFS client libraries, so GKE gets around this by setting up a chroot (at /home/kubernetes/containerized_mounter/rootfs) with the required binaries and calling mount inside that.

Answer 2

I've nicked the solution @saad-ali mentioned above, from the kubernetes project, to make this work.

To be concrete, I've added the following to my cloud-config:

# This script creates a chroot environment containing the tools needed to mount an nfs drive
- path: /tmp/mount_config.sh
  permissions: 0755
  owner: root
  content: |
    #!/bin/sh
    set +x # For debugging

    export USER=root
    export HOME=/home/dockerrunner
    mkdir -p /tmp/mount_chroot
    chmod a+x /tmp/mount_chroot
    cd /tmp/
    echo "Sleeping for 30 seconds because toolbox pull fails otherwise"
    sleep 30
    toolbox --bind /tmp /google-cloud-sdk/bin/gsutil cp gs://<uploaded-file-bucket>/mounter.tar /tmp/mounter.tar
    tar xf /tmp/mounter.tar -C /tmp/mount_chroot/
    mount --bind /tmp/mount_chroot /tmp/mount_chroot
    mount -o remount, exec /tmp/mount_chroot
    mount --rbind /proc /tmp/mount_chroot/proc
    mount --rbind /dev /tmp/mount_chroot/dev
    mount --rbind /tmp /tmp/mount_chroot/tmp
    mount --rbind /mnt /tmp/mount_chroot/mnt

The uploaded-file-bucket container the chroom image the kube team has created, downloaded from: https://storage.googleapis.com/kubernetes-release/gci-mounter/mounter.tar

Then, the runcmd for the cloud-config looks something like:

runcmd:
- /tmp/mount_config.sh
- mkdir -p /mnt/disks/nfs_mount
- chroot /tmp/mount_chroot /bin/mount -t nfs -o rw nfsserver:/sftp /mnt/disks/nfs_mount

This works. Ugly as hell, but it'll have to do for now.