Run MongoDB in Kubernetes with Keyfile

Question

i try to run a mongodb within a kubernetes cluster secured with a keyFile. For this, i created a simple statefulset and a configmap, where i stored the keyfile:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  serviceName: mongodb
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:4.4
        args:
          - --bind_ip
          - '0.0.0.0,::'
          - --replSet
          - MySetname01
          - --auth
          - --keyFile
          - /etc/mongodb/keyfile/keyfile
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          value: MyUsername
        - name: MONGO_INITDB_ROOT_PASSWORD
          value: MyPassword
        ports:
        - containerPort: 27017
          name: mongodb
        volumeMounts:
        - name: mongodb-persistent-storage
          mountPath: /data/db
        - name: mongodb-keyfile
          mountPath: /etc/mongodb/keyfile
          readOnly: True
      volumes:
        - name: mongodb-keyfile
          configMap:
            name: mongodb-keyfile
  volumeClaimTemplates:
  - metadata:
      name: mongodb-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  labels:
    app: mongodb
spec:
  ports:
    - port: 27017
  selector:
    app: mongodb
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-keyfile
data:
  keyfile: |
    +PN6gXEU8NeRsyjlWDnTesHCoPOn6uQIEI5pNorDkphREi6RyoSHCIaXOzLrUpPq
    jpSGhSc5/MZj17R7K5anjerhvR6f5JtWjBuQcrjdJdNBceck71F2bly/u9ICfCOy
    STFzv6foMQJBJTBYqLwtfyEO7CQ9ywodM0K5r9jtT7x5BiJaqso+F8VN/VFtIYpe
    vnzKj7uU3GwDbmw6Yduybgv6P88BGXyW3w6HG8VLMgud5aV7wxIIPE6nAcr2nYmM
    1BqC7wp8G6uCcMiHx5pONPA5ONYAIF+u3zj2wAthgMe2UeQxx2L2ERx8Zdsa9HLR
    qYOmy9XhfolwdCTwwYvqYRO+RqXGoPGczenC/CKJPj14yfkua+0My5NBWvpL/fIB
    osu0lQNw1vFu0rcT1/9OcaJHuwFWocec2qBih9tk2C3c7jNMuxkPo3dxjv8J/lex
    vN3Et6tK/wDsQo2+8j6uLYkPFQbHZJQzf/oQiekV4RaC6/pejAf9fSAo4zbQXh29
    8BIMpRL3fik+hvamjrtS/45yfqGf/Q5DQ7o8foI4HYmhy+SU2+Bxyc0ZLTn659zl
    myesNjB6uC9lMWtpjas0XphNy8GvJxfjvz+bckccPUVczxyC3QSEIcVMMH9vhzes
    AcQscswhFMgzp1Z0fbNKy0FqQiDy1hUSir06ZZ3xBGLKeIySRsw9D1Pyh1Y11HlH
    NdGwF14cLqm53TGVd9gYeIAm2siQYMKm8rEjxmecc3yGgn0B69gtMcBmxr+z3xMU
    X256om6l8L2BJjm3W1zUTiZABuKzeNKjhmXQdEFPQvxhubvCinTYs68XL76ZdVdJ
    Q909MmllkOXKbAhi/TMdWmpV9nhINUCBrnu3F08jAQ3UkmVb923XZBzcbdPlpuHe
    Orp11/f3Dke4x0niqATccidRHf6Hz+ufVkwIrucBZwcHhK4SBY/RU90n233nV06t
    JXlBl/4XjWifB7iJi9mxy/66k

Problem is: MongoDb stays in a Crashloopbackoff , because the permissions on the keyfile are too open:

{"t":{"$date":"2022-12-19T12:41:41.399+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-12-19T12:41:41.402+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-12-19T12:41:41.402+00:00"},"s":"I", "c":"ACCESS", "id":20254, "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"permissions on /etc/mongodb/keyfile/keyfile are too open"}}}

For what i dont have a explanation.

I already set the volumemount of the configmap on readonly (You see in the mongo statefulset). Also i tried around with commands or lifecyclehooks to chmod 600/400 the file. I tried differnt versions of mongodb, but got always the same error. For sure i also tried if the configmap is included correctly, it is (I uncommented the args and Username/Password for that one).

Permissions are shown:

lrwxrwxrwx 1 root root   14 Dec 19 12:50 keyfile -> ..data/keyfile

Maybe its related to that fact that the file is shown as linked?

I expect a kubernetes yaml which is able to start with a keyfile. Thank you very much.

---

EDIT: I tried to mount the file directly, not as a link with subpath. Now i got the following permissions:

-rw-r--r-- 1 root root 1001 Dec 19 13:34 mongod.key

But sadly the db will not start with that one too, it's still crashing with the same error.

---

EDIT2: Adding defaultMode: 0600 to the volume in the statefulset led at least to the correct permissions, but also another error (already mentioned in one of my comments):

file: /var/lib/mongo/mongod.key: bad file"

So i tried to mount on different places in the Pod (You see here /var/lib/) for example and i tried to include the keyfile as secret. But none is working.

Answer 1

this works for me:

ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: common-mongodb
data:
  keyfile: |
    <hash>

StatefulSet

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: common-mongodb
    namespace: default
spec:
  selector:
    matchLabels:
      app: common-mongodb
  serviceName: "common-mongodb"
  podManagementPolicy: Parallel
  replicas: 1
  template:
    metadata:
      labels:
        app: common-mongodb
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: common-mongodb
        image: mongo
        imagePullPolicy: IfNotPresent
        args:
          - --bind_ip
          - '0.0.0.0,::'
          - --replSet
          - rpl0
          - --auth
          - --keyFile
          - /etc/mongo/keyfile
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          value: user
        - name: MONGO_INITDB_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 27017
          name: common-mongodb
        volumeMounts:
        - name: common-mongodb-data
          mountPath: /etc/mongo
      initContainers:
      - name: change-ownership-container
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh"]
        args:
          - -c
          - >-
              /bin/cp /tmp/keyfile /etc/mongo/keyfile &&
              /bin/chown 999:999 /etc/mongo/keyfile &&
              /bin/chmod 400 /etc/mongo/keyfile
        volumeMounts:
        - name: common-mongodb
          mountPath: /tmp/
        - name: common-mongodb-data
          mountPath: /etc/mongo
      volumes:
        - name: common-mongodb
          defaultMode: 0400
          configMap:
            name: common-mongodb
  volumeClaimTemplates:
  - metadata:
      name: common-mongodb-data
      namespace: default
    spec:
      storageClassName: rook-ceph-block
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Gi

Maybe not the best solution, but it works

Answer 2

If anyone is still looking for a solution to this issue, I found one that does not use the init container.

Working with k8s 1.26, I mounted the keyfile as a secret inside the pod with a volume and I set the defaultMode to 0o400.

We have to put the "o" after the "0" so the system would recognize it as readonly for the owner and we will be able to get the file with a permission like so -r--------.

I mounted the secret in /var/run/secrets/keyfile and using a subPath: ./keyfile

Here is the statefulset yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongod
  namespace: test
spec:
  serviceName: mongodb-service
  replicas: 1
  selector:
    matchLabels:
      role: mongo
      app: mongo
      replicaset: rsTest
  template:
    metadata:
      labels:
        role: mongo
        app: mongo
        replicaset: rsTest
    spec:
      volumes:
        - name: mongo-conf
          configMap:
            name: mongo-conf-cm
        - name: mongodb-keyfile
          secret:
            secretName: mongodb-keyfile
            defaultMode: 0o400
      containers:
      - name: mongod-container
        image: mongo:6.0.1
        command:
        - "numactl"
        - "--interleave=all"
        - "mongod"
        - "--config"
        - "/etc/mongo.conf"
        resources:
          requests:
            cpu: 0.2
            memory: 200Mi
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongodb-persistent-storage-claim
          mountPath: /data/db
        - name: mongo-conf
          mountPath: /etc/mongo.conf
          subPath: mongo.conf
        - name: mongodb-keyfile
          mountPath: /var/run/secrets/keyfile
          subPath: ./keyfile
          readOnly: true
  volumeClaimTemplates:
  - metadata:
      name: mongodb-persistent-storage-claim
      annotations:
        volume.beta.kubernetes.io/storage-class: "standard"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi