mkdir /mnt/data: read-only file system Back-off restarting failed postgres container

Question

I'm new to Kubernetes, I tried to apply yaml file to create Postgres in GKE, I'm getting error as "Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system Back-off restarting failed container.

I thinki need to give permsions as RWX , when i tried to Login to pod i.e inside container..It is not allowing to login. ANyone please help me !!.

This is my Yaml file for Postgres:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: postgres
    spec:
      selector:
       matchLabels:
        app: postgres
      replicas: 1
      template:
        metadata:
          labels:
            app: postgres
        spec:
          containers:
            - name: postgres
              image: postgres:latest
              imagePullPolicy: "IfNotPresent"
              envFrom:
                - configMapRef:
                    name: postgres-config
              volumeMounts:
                - mountPath: /var/lib/postgresql/data
                  name: postgredb
          volumes:
            - name: postgredb
              persistentVolumeClaim:
                claimName: postgres-pv-claim

---
    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: postgres-pv-volume
      labels:
        type: local
        app: postgres
    spec:
      storageClassName: manual
      capacity:
        storage: 5Gi
      accessModes:
        - ReadWriteMany
      hostPath:
        path: "/mnt/data"
---
   

     kind: PersistentVolumeClaim
        apiVersion: v1
        metadata:
          name: postgres-pv-claim
          labels:
            app: postgres
        spec:
          storageClassName: manual
          accessModes:
            - ReadWriteMany
          resources:
            requests:
              storage: 5Gi
        
    ---
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: postgres-config
          labels:
            app: postgres
        data:
          POSTGRES_DB: postgresdb
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: root
    ---
        apiVersion: v1
        kind: Service
        metadata:
          name: postgres
          labels:
            app: postgres
        spec:
           ports:
            - name: postgres
              port: 5432
              nodePort: 30432
           type: NodePort
           selector:
            app: postgres

Answer 1

In your Persistent Volume you are using type: local which means that you want to create directory in /mnt. Local also do not support dynamic volume provisioning. If you will SSH to any of your nodes you will find that this folder is ReadOnly file system.

/mnt $ mkdir something mkdir: cannot create directory ‘something’: Read-only file system

As fastest workaround, you just could change in your PV YAML

    - ReadWriteMany
  hostPath:
    path: /mnt/data

To:

    - ReadWriteMany
  hostPath:
    path: /var/lib/data

Example:

$ kubectl apply -f pv-pvc.yaml
persistentvolume/postgres-pv-volume created
persistentvolumeclaim/postgres-pv-claim created
$ kubectl apply -f pos.yaml
deployment.apps/postgres created
$ kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
postgres-65d9cbd495-pcqf5   1/1     Running   0          2s

$ kubectl exec -ti postgres-65d9cbd495-pcqf5 -- /bin/bash
root@postgres-65d9cbd495-pcqf5:/# cd /var/lib/postgresql/data
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# ls
base    pg_commit_ts  pg_hba.conf    pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_wal   postgresql.auto.conf  postmaster.opts
global  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   pg_xact  postgresql.conf       postmaster.pid
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# echo "Hello from postgress pod" > data.txt
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# cat data.txt
Hello from postgress pod

Now if you will SSH to the node which is hosting this pod, you will be able to reach this folder and files.

user@gke-cluster-1-default-pool-463f9615-gxhl ~ $ sudo su
gke-cluster-1-default-pool-463f9615-gxhl /home/user # cd /var/lib/data
gke-cluster-1-default-pool-463f9615-gxhl /var/lib/data # ls
PG_VERSION    pg_dynshmem    pg_notify     pg_stat_tmp  pg_xact
base          pg_hba.conf    pg_replslot   pg_subtrans  postgresql.auto.conf
data.txt      pg_ident.conf  pg_serial     pg_tblspc    postgresql.conf
global        pg_logical     pg_snapshots  pg_twophase  postmaster.opts
pg_commit_ts  pg_multixact   pg_stat       pg_wal       postmaster.pid
gke-cluster-1-default-pool-463f9615-gxhl /var/lib/data # cat data.txt 
Hello from postgress pod

EDIT

YAMLs Ive used.

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: root
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
  - name: postgres
    port: 5432
    nodePort: 30432
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume
  labels:
    app: postgres
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /var/lib/data
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim
            
configmap/postgres-config created
service/postgres created
persistentvolume/postgres-pv-volume created
persistentvolumeclaim/postgres-pv-claim created
deployment.apps/postgres created

$ kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
postgres-65d9cbd495-wxx4h   1/1     Running   0          19s

Answer 2

If you're working with GKE, just create PVC, it will self create PV which will work according to your need.

I Fixed my issue that way.