Standalone kubelet persistent disk mount on Google Cloud instance

Question

I have a process that needs to run daily in a Docker container, syncing some data from a storage bucket to an external volume (Google Cloud persistent disk). So far, I managed to launch the process by creating a single-node container cluster.

Since the process completes in a couple of hours, I want to delete VM resources (except the persistent disk of course) once complete. Launching/deleting a single compute-VM (without the kubernetes cluster setup) seems simpler, so I was trying to get a single kubelet running on a container-optimized cloud instance. Persistent disk mounting is where this fails.

My launch command:

gcloud compute instances create cvm-name-0 \
    --image-family=cos-stable \
    --image-project=cos-cloud \
    --boot-disk-type pd-ssd \
    --boot-disk-size 10GB \
    --metadata-from-file \
    "google-container-manifest=containers.yaml,user-data=cloudinit.yaml" \
    --zone "$gzone" \
    --scopes default,storage-rw \
    --machine-type n1-highcpu-4

Contents of container.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: container-name
spec:
  containers:
    - name: container-name
      image: gcr.io/project-name/container-name
      imagePullPolicy: Always
      volumeMounts:
        - name: persistent-disk-name
          mountPath: /home/someuser/somedir
  volumes:
    - name: persistent-disk-name
      gcePersistentDisk:
        pdName: persistent-disk-name
        fsType: ext4

Contents of cloudinit.yaml:

#cloud-config
bootcmd:
- echo "KUBELET_OPTS=\"--cloud-provider=gce\"" > /etc/default/kubelet
runcmd:
- systemctl start kubelet.service

While the --cloud-provider=gce option fixes the "Failed to get GCE Cloud Provider" error per this question, there is still some problem mounting the disk.

A potentially relevant line from the container OS log says:

EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities

Any way to make this work on a single compute instance (without the kubernetes cluster)? Where else should I be looking for more informative error logs?

Answer 1

I'm not using kubernetes at the moment, but I am backing up to a cloud storage bucket.

I have something like this in my cloud-config:

users:
- name: dockerrunner
  uid: 2000
  groups: docker
write_files:
- path: /home/dockerrunner/backup-hourly.sh
  permissions: 0755
  owner: dockerrunner
  content: |
    #!/bin/sh
    export HOME=/home/dockerrunner
    export USER=root
    toolbox --bind /mnt/disks/nfs:/mnt/disks/nfs \
      /google-cloud-sdk/bin/gsutil -m \
      rsync -r /mnt/disks/nfs gs://<bucket-name>/hourly
- path: /etc/systemd/system/files-backup-hourly.service
  permissions: 0644
  owner: root
  content: |
    [Unit]
    Description=Shared Files Backup upload script - hourly

    [Service]
    Type=oneshot
    ExecStart=/bin/sh /home/dockerrunner/backup-hourly.sh

- path: /etc/systemd/system/files-backup-hourly.timer
  permissions: 0644
  owner: root
  content: |
    [Unit]
    Description=Run Shared Files Backup create script every hour

    [Timer]
    OnCalendar=*-*-* *:00:00

    [Install]
    WantedBy=timers.target
runcmd:
- systemctl daemon-reload
- systemctl start files-backup-hourly.service
- systemctl start files-backup-hourly.timer