List Kubernetes containers that use docker local volumes

Question

When deploying a workload that has a VOLUME in a Dockerfile, that volume may not be mapped to a persistent volume (PV/PVC) in Kubernetes.

Actually, unless a Kubernetes volume is attached to that workload, The docker daemon container will temporarily create a docker-volume when starting the container (driver type: local). Kubernetes won't be aware of it. See: are VOLUME in Dockerfile persistent in kubernetes. This docker volume will be destroyed when the pod is removed or redeployed.

It is certainly good practice to use a kubernetes volume, even ephermeal volumes (or generic ephemeral volumes... still in alpha in 1.19)

Q: How to list pods/containers that use such local volumes?

This is really important since restarting the workload/deployment/stateful-set will cause disruption (lost of ephemeral volume).

Answer 1

Here's a little script you can run on the Kubernetes nodes (for installation that use Docker daemon).

for d in $(docker volume ls -q --filter driver=local ) ; do
  echo "=== volume $d "
  CID=$(docker ps -q -a --filter volume=$d)
  docker container inspect  -f '  Namespace : {{ index .Config.Labels "io.kubernetes.pod.namespace"}}{{ printf "\n" }}  contnr ID : {{.ID}}{{ printf "\n" }}  Name      : {{.Name}}{{ printf "\n" }}{{ range .Mounts }}{{if eq .Driver "local" }}  Mountpoint: {{ .Destination }}  (RW:{{.RW}}){{end}}{{ end }}'  $CID
  du -ms /var/lib/docker/volumes/$d/_data | sed -e 's/^\([0-9]\+\).*/  Size      : \1 MiB/';
  echo
  echo
done

exemple:

=== volume c975149a17753393c543e25e4391af849d14d6d0cf2db4f4e873901ff05fea96
  Namespace :
  contnr ID : 01457a869b5a849952710a1bb023b10cf1f2dbf5779326d9c39a409b127a1437
  Name      : /nginx-proxy
  Mountpoint: /opt/rke-tools  (RW:true)
  Size      : 25 MiB


=== volume f8dbc7f9f06e82253ee882e19b72cbef5b30ba134230543aa34605cafcb6b082
  Namespace : foo
  contnr ID : 71758f3de8ab04b8547554eaf43e9ff54410feef597c55e628b8a3bb19c50e12
  Name      : /k8s_pgadmin4_foo-bar-001-pgadmin4-6d867f44d8-t566l_foo-bar_36715dd4-b858-4e8e-bc1c-c65cd7004e41_0
  Mountpoint: /var/lib/pgadmin  (RW:true)
  Size      : 1 MiB

Note that some system pods/containers don't have namespace.

This script should be ran on each node you want to audit.

An easy way to scan multiple host from a central server using ssh, would be to copy the script above in a file local_volumes.sh, then execute a command like cat local_volumes.sh | ssh node001 sudo bash -

For rancher users, this snippet audit all cordoned nodes: for s in $(rancher nodes ls | grep cordoned | cut -d " " -f 1 ); do cat local_volumes.sh | rancher ssh $s sudo bash - ;done