Subdirectory of PV directory isn't mounted to persistent volume

Question

I've got a PV that is backed by do-block-storage, and it's mounted to a stateful set, at /var/lib/postgresql, this is the full stateful set:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: psql-api-set
spec:
  selector:
    matchLabels:
      app: psql-api
  serviceName: "psql-api"
  replicas: 1
  template:
    metadata:
      labels:
        app: psql-api
    spec:
      containers:
        - image: postgres:13.2
          name: db
          ports:
            - containerPort: 5432
              name: psql-api
          volumeMounts:
            - name: psql-api-v3-persistent-storage
              mountPath: /var/lib/postgresql
      initContainers:
      - name: pgsql-data-permission-fix
        image: busybox
        command: ["/bin/chmod","-R","777", "/data"]
        volumeMounts:
        - name: psql-api-v3-persistent-storage
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: psql-api-v3-persistent-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: do-block-storage

This all works, for the most part, I can cd into the pod and see the mounted directory:

df -h /var/lib/postgresql
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-e0dedf16-188d-4524-b524-d4347433ebd7  9.8G   37M  9.3G   1% /var/lib/postgresql

The problem is that the directory /var/lib/postgresql/data is using the local filesystem, not the mounted persistent volume:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       158G   30G  122G  20% /var/lib/postgresql/data

So when the server restarts, the data isn't persisted, because that directory isn't on the persistent volume.

I would have assumed that since /var/lib/postgresql is mounted from the persistent volume, that when postgres initializes the data, it would also be on the persistent volume, but it appears this isn't the case. Any help would be greatly appreciated, thank you.

Here's the full output of mount:

overlay on / type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/376/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/375/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/373/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/372/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/371/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/347/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/344/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/337/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/335/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/327/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/325/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/324/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/321/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/289/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/377/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/377/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,errors=remount-ro)
/dev/vda1 on /dev/termination-log type ext4 (rw,relatime,errors=remount-ro)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro)
/dev/disk/by-id/scsi-0DO_Volume_pvc-e0dedf16-188d-4524-b524-d4347433ebd7 on /var/lib/postgresql type ext4 (rw,relatime)
/dev/vda1 on /var/lib/postgresql/data type ext4 (rw,relatime,errors=remount-ro)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /sys/firmware type tmpfs (ro,relatime)

Alright I thought maybe it was the initContainers thing that was messing it up, but even if I remove that, delete the stateful set, recreate the storage by giving it a new name, AND change the mount path to /data, the postgres directory still gets created as a mount to /dev/vda instead of the persistent storage:

/dev/disk/by-id/scsi-0DO_Volume_pvc-134861c1-cd02-4937-9a1a-440fce4d72bb on /data type ext4 (rw,relatime)
/dev/vda1 on /var/lib/postgresql/data type ext4 (rw,relatime,errors=remount-ro)

If I set the mount path to /data/pgdata, and set PGDATA to that same directory, I get:

initdb: error: directory "/data/pgdata" exists but is not empty
It contains a lost+found directory, perhaps due to it being a mount point.
Using a mount point directly as the data directory is not recommended.