CephFS Unable to attach or mount volumes: unmounted volumes=[image-store]

Question

I'm having trouble getting my Kube-registry up and running on cephfs. I'm using rook to set this cluster up. As you can see, I'm having trouble attaching the volume. Any idea what would be causing this issue? any help is appreciated.

kube-registry.yaml

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: cephfs-pvc
      namespace: kube-system
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 1Gi
      storageClassName: rook-cephfs
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kube-registry
      namespace: kube-system
      labels:
        k8s-app: kube-registry
        kubernetes.io/cluster-service: "true"
    spec:
      replicas: 3
      selector:
        matchLabels:
          k8s-app: kube-registry
      template:
        metadata:
          labels:
            k8s-app: kube-registry
            kubernetes.io/cluster-service: "true"
        spec:
          containers:
          - name: registry
            image: registry:2
            imagePullPolicy: Always
            resources:
              limits:
                cpu: 100m
                memory: 100Mi
            env:
            # Configuration reference: https://docs.docker.com/registry/configuration/
            - name: REGISTRY_HTTP_ADDR
              value: :5000
            - name: REGISTRY_HTTP_SECRET
              value: "Ple4seCh4ngeThisN0tAVerySecretV4lue"
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: /var/lib/registry
            volumeMounts:
            - name: image-store
              mountPath: /var/lib/registry
            ports:
            - containerPort: 5000
              name: registry
              protocol: TCP
            livenessProbe:
              httpGet:
                path: /
                port: registry
            readinessProbe:
              httpGet:
                path: /
                port: registry
          volumes:
          - name: image-store
            persistentVolumeClaim:
              claimName: cephfs-pvc
              readOnly: false

Storagelass.yaml

        apiVersion: storage.k8s.io/v1
        kind: StorageClass
        metadata:
          name: rook-cephfs
        # Change "rook-ceph" provisioner prefix to match the operator namespace if needed
        provisioner: rook-ceph.cephfs.csi.ceph.com
        parameters:
          # clusterID is the namespace where operator is deployed.
          clusterID: rook-ceph
          # CephFS filesystem name into which the volume shall be created
          fsName: myfs
          # Ceph pool into which the volume shall be created
          # Required for provisionVolume: "true"
          pool: myfs-data0
          # Root path of an existing CephFS volume
          # Required for provisionVolume: "false"
          # rootPath: /absolute/path
          # The secrets contain Ceph admin credentials. These are generated automatically by the operator
          # in the same namespace as the cluster.
          csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
          csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
          csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
          csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
        reclaimPolicy: Deletea

kubectl describe pods --namespace=kube-system kube-registry-58659ff99b-j2b4d

        Name:           kube-registry-58659ff99b-j2b4d
        Namespace:      kube-system
        Priority:       0
        Node:           minikube/192.168.99.212
        Start Time:     Wed, 25 Nov 2020 13:19:35 -0500
        Labels:         k8s-app=kube-registry
                        kubernetes.io/cluster-service=true
                        pod-template-hash=58659ff99b
        Annotations:    <none>
        Status:         Pending
        IP:
        IPs:            <none>
        Controlled By:  ReplicaSet/kube-registry-58659ff99b
        Containers:
          registry:
            Container ID:
            Image:          registry:2
            Image ID:
            Port:           5000/TCP
            Host Port:      0/TCP
            State:          Waiting
              Reason:       ContainerCreating
            Ready:          False
            Restart Count:  0
            Limits:
              cpu:     100m
              memory:  100Mi
            Requests:
              cpu:      100m
              memory:   100Mi
            Liveness:   http-get http://:registry/ delay=0s timeout=1s period=10s #success=1 #failure=3
            Readiness:  http-get http://:registry/ delay=0s timeout=1s period=10s #success=1 #failure=3
            Environment:
              REGISTRY_HTTP_ADDR:                         :5000
              REGISTRY_HTTP_SECRET:                       Ple4seCh4ngeThisN0tAVerySecretV4lue
              REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /var/lib/registry
            Mounts:
              /var/lib/registry from image-store (rw)
              /var/run/secrets/kubernetes.io/serviceaccount from default-token-nw4th (ro)
        Conditions:
          Type              Status
          Initialized       True
          Ready             False
          ContainersReady   False
          PodScheduled      True
        Volumes:
          image-store:
            Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
            ClaimName:  cephfs-pvc
            ReadOnly:   false
          default-token-nw4th:
            Type:        Secret (a volume populated by a Secret)
            SecretName:  default-token-nw4th
            Optional:    false
        QoS Class:       Guaranteed
        Node-Selectors:  <none>
        Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                         node.kubernetes.io/unreachable:NoExecute for 300s
        Events:
          Type     Reason              Age                 From                     Message
          ----     ------              ----                ----                     -------
          Warning  FailedScheduling    13m (x3 over 13m)   default-scheduler        running "VolumeBinding" filter plugin for pod "kube-registry-58659ff99b-j2b4d": pod has unbound immediate PersistentVolumeClaims
          Normal   Scheduled           13m                 default-scheduler        Successfully assigned kube-system/kube-registry-58659ff99b-j2b4d to minikube
          Warning  FailedMount         2m6s (x5 over 11m)  kubelet, minikube        Unable to attach or mount volumes: unmounted volumes=[image-store], unattached volumes=[image-store default-token-nw4th]: timed out waiting for the condition
          Warning  FailedAttachVolume  59s (x6 over 11m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6eeff481-eb0a-4269-84c7-e744c9d639d9" : attachdetachment timeout for volume 0001-0009-rook-c

ceph provisioner logs, I restarted my cluster so the name will be different but output is the same

        I1127 18:27:19.370543       1 csi-provisioner.go:121] Version: v2.0.0
        I1127 18:27:19.370948       1 csi-provisioner.go:135] Building kube configs for running in cluster...
        I1127 18:27:19.429190       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
        I1127 18:27:21.561133       1 common.go:111] Probing CSI driver for readiness
        W1127 18:27:21.905396       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
        I1127 18:27:22.060963       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/rook-ceph-cephfs-csi-ceph-com...
        I1127 18:27:22.122303       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
        I1127 18:27:22.323990       1 controller.go:820] Starting provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-797b67c54b-42jwc_4e14295b-f73d-4b94-bae9-ff4f2639b487!
        I1127 18:27:22.324061       1 clone_controller.go:66] Starting CloningProtection controller
        I1127 18:27:22.324205       1 clone_controller.go:84] Started CloningProtection controller
        I1127 18:27:22.325240       1 volume_store.go:97] Starting save volume queue
        I1127 18:27:22.426790       1 controller.go:869] Started provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-797b67c54b-42jwc_4e14295b-f73d-4b94-bae9-ff4f2639b487!
        I1127 19:08:39.850493       1 controller.go:1317] provision "kube-system/cephfs-pvc" class "rook-cephfs": started
        I1127 19:08:39.851034       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kube-system", Name:"cephfs-pvc", UID:"7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06", APIVersion:"v1", ResourceVersion:"7744", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "kube-system/cephfs-pvc"
        I1127 19:08:43.670226       1 controller.go:1420] provision "kube-system/cephfs-pvc" class "rook-cephfs": volume "pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06" provisioned
        I1127 19:08:43.670262       1 controller.go:1437] provision "kube-system/cephfs-pvc" class "rook-cephfs": succeeded
        E1127 19:08:43.692108       1 controller.go:1443] couldn't create key for object pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06: object has no meta: object does not implement the Object interfaces
        I1127 19:08:43.692189       1 controller.go:1317] provision "kube-system/cephfs-pvc" class "rook-cephfs": started
        I1127 19:08:43.692205       1 controller.go:1326] provision "kube-system/cephfs-pvc" class "rook-cephfs": persistentvolume "pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06" already exists, skipping
        I1127 19:08:43.692220       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kube-system", Name:"cephfs-pvc", UID:"7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06", APIVersion:"v1", ResourceVersion:"7744", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned 

Answer 1

In the pasted YAML for your StorageClass, you have:

reclaimPolicy: Deletea

Was that a paste issue? Regardless, this is likely what is causing your problem.

I just had this exact problem with some of my Ceph RBD volumes, and the reason for it was that I was using a StorageClass that had

reclaimPolicy: Delete

However, the cephcsi driver was not configured to support it (and I don't think it actually supports it either).

Using a StorageClass with

reclaimPolicy: Retain

fixed the issue.

To check this on your cluster, run the following:

$ kubectl get sc rook-cephfs -o yaml

And look for the line that starts with reclaimPolicy:

Then, look at the csidriver your StorageClass is using. In your case it is rook-ceph.cephfs.csi.ceph.com

$ kubectl get csidriver rook-ceph.cephfs.csi.ceph.com -o yaml

And look for the entries under volumeLifecycleModes

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  creationTimestamp: "2020-11-16T22:18:55Z"
  name: rook-ceph.cephfs.csi.ceph.com
  resourceVersion: "29863971"
  selfLink: /apis/storage.k8s.io/v1beta1/csidrivers/rook-ceph.cephfs.csi.ceph.com
  uid: a9651d30-935d-4a7d-a7c9-53d5bc90c28c
spec:
  attachRequired: true
  podInfoOnMount: false
  volumeLifecycleModes:
  - Persistent

If the only entry under volumeLifecycleModes is Persistent, then your driver is not configured to support reclaimPolicy: Delete.

If instead you see

volumeLifecycleModes:
    - Persistent
    - Ephemeral

Then your driver should support reclaimPolicy: Delete