GKE gke filestore csi failed when mouted to pod: Unable to attach or mount volumes: timed out waiting for the condition

Question

I get error Unable to attach or mount volumes: unmounted volumes=[file-store], unattached volumes=[file-store kube-api-access-p7btw]: timed out waiting for the condition.

 kubectl describe pod kickstar-backend-5577cf96cf-rlpht
Name:             kickstar-backend-5577cf96cf-rlpht
Namespace:        default
Priority:         0
Service Account:  default
Node:             gke-kickstar-prod-workloads-clus-main-ab940238-1ktc/10.128.0.5
Start Time:       Tue, 08 Aug 2023 03:31:56 +0000
Labels:           app=kickstar-backend
                  pod-template-hash=5577cf96cf
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/kickstar-backend-5577cf96cf
Containers:
  kickstar-backend:
    Container ID:   
    Image:          asia-southeast1-docker.pkg.dev/kickstar-prod/kickstar/kickstar.backend:13fb26c
    Image ID:       
    Port:           1026/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /app/wwwroot from file-store (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p7btw (ro)
Readiness Gates:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
Conditions:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
  Initialized                                True 
  Ready                                      False 
  ContainersReady                            False 
  PodScheduled                               True 
Volumes:
  file-store:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  file-store-pvc
    ReadOnly:   false
  kube-api-access-p7btw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                   Age                From                     Message
  ----     ------                   ----               ----                     -------
  Normal   LoadBalancerNegNotReady  27m (x2 over 27m)  neg-readiness-reflector  Waiting for pod to become healthy in at least one of the NEG(s): [k8s1-0912e870-default-kickstar-backend-80-4d4b640a]
  Normal   NotTriggerScaleUp        27m                cluster-autoscaler       pod didn't trigger scale-up:
  Warning  FailedScheduling         24m (x2 over 27m)  default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled                24m                default-scheduler        Successfully assigned default/kickstar-backend-5577cf96cf-rlpht to gke-kickstar-prod-workloads-clus-main-ab940238-1ktc
  Warning  FailedMount              21m (x6 over 22m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-9cd5a129-d237-4074-95ae-de42446fc75c" : rpc error: code = Aborted desc = An operation with the given volume key modeInstance/asia-southeast1-a/pvc-9cd5a129-d237-4074-95ae-de42446fc75c/vol1 already exists.
 --- Most likely a long process is still running to completion. Retrying.
  Normal   LoadBalancerNegTimeout  12m                  neg-readiness-reflector  Timeout waiting for pod to become healthy in at least one of the NEG(s): [k8s1-0912e870-default-kickstar-backend-80-4d4b640a]. Marking condition "cloud.google.com/load-balancer-neg-ready" to True.
  Warning  FailedMount             4m12s (x2 over 11m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[file-store], unattached volumes=[kube-api-access-p7btw file-store]: timed out waiting for the condition
  Warning  FailedMount             118s (x8 over 22m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[file-store], unattached volumes=[file-store kube-api-access-p7btw]: timed out waiting for the condition
  Warning  FailedMount             5s (x7 over 22m)     kubelet                  MountVolume.MountDevice failed for volume "pvc-9cd5a129-d237-4074-95ae-de42446fc75c" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

The PVs and PVCs are all seemed to be successfully applied.

kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
persistentvolume/pvc-9cd5a129-d237-4074-95ae-de42446fc75c   1Ti        RWX            Delete           Bound    default/file-store-pvc   file-store              31m

NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/file-store-pvc   Bound    pvc-9cd5a129-d237-4074-95ae-de42446fc75c   1Ti        RWX            file-store     34m

Following are the configs for SC, PV and PVC:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: file-store
provisioner: filestore.csi.storage.gke.io
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  tier: standard
  network: default
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: file-store-pvc
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: file-store
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kickstar-backend
spec:
  replicas: 1
  revisionHistoryLimit: 4
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - image: backend:latest
        name: backend
        ports:
        - containerPort: 1026
        volumeMounts:
        - mountPath: /app/wwwroot
          name: file-store
      volumes:
      - name: file-store
        persistentVolumeClaim:
          claimName: file-store-pvc

Can anyone help me where I'm wrong exactly? Thanks in advance and best regards.

Can anyone help me where I'm wrong exactly? Thanks in advance and best regards.