In my Kubernetes cluster, I tried to start a deployment of MariaDB with 3 replicas. After 30 s 2/3 replicas crash, restart, and crashes again. It has something to do with the PVC which is linked to the Deployment. There are some indications that the pods need an exclusive look at some of the files in the PV. One of the pods works as expected.
Similar setups work for both influxDB and Grafana. But something is happening with the MariaDB.
The .yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
labels:
app: mariadb
spec:
replicas: 3
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- name: mariadb
image: mariadb:latest
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: MYSQL_ROOT_PASSWORD
name: platform-secret
[more variables]
volumeMounts:
- mountPath: /var/lib/mysql
name: mariadb-pv
restartPolicy: Always
volumes:
- name: mariadb-pv
persistentVolumeClaim:
claimName: mariadb-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mariadb-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
labels:
app: mariadb
name: mariadb
spec:
selector:
app: influxdb
type: ClusterIP
ports:
- protocol: TCP
port: 3306
targetPort: 3306
status:
loadBalancer: {}
I tried to divide the PVC and added to volumes for subfolders, which worked fine:
volumeMounts:
- name: mariadb-aria-storage
mountPath: /var/lib/mysql/aria
- name: mariadb-innodb-storage
mountPath: /var/lib/mysql/innodb
So something is wrong with the PVC when mapping /var/lib/mysql
Some error indicators:
kubectl get pods
NAME READY STATUS RESTARTS AGE
mariadb-78d7945894-gsjnq 1/1 Running 2 (28s ago) 103s
mariadb-78d7945894-qzfqd 1/1 Running 0 103s
mariadb-78d7945894-zmg7k 1/1 Running 2 (27s ago) 103s
kubectl get event
[...]
2m10s Warning FailedAttachVolume pod/mariadb-78d7945894-zmg7k AttachVolume.Attach failed for volume "pvc-c5b09340-5060-4b77-921a-9f889bc81e84" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach e162b4b1-be0f-4b63-a8eb-6bf679c9719e volume to cfb3d471-5184-4a79-8474-dc62c4893c7d compute: Bad request with: [POST https://ops.elastx.cloud:8774/v2.1/servers/cfb3d471-5184-4a79-8474-dc62c4893c7d/os-volume_attachments], error message: {"badRequest": {"code": 400, "message": "Invalid input received: Invalid volume: Volume e162b4b1-be0f-4b63-a8eb-6bf679c9719e status must be available or downloading to reserve, but the current status is creating. (HTTP 400) (Request-ID: req-6f2d38f9-c0d4-43ab-81bf-da75d347e723)"}}
2m17s Warning FailedScheduling pod/mariadb-78d7945894-zmg7k running PreBind plugin "VolumeBinding": Operation cannot be fulfilled on persistentvolumeclaims "mariadb-pvc": the object has been modified; please apply your changes to the latest version and try again
1s Warning BackOff pod/mariadb-78d7945894-zmg7k Back-off restarting failed container mariadb in pod mariadb-78d7945894-zmg7k_default(d0952758-94e9-4d9e-aeda-c8f851a891b6)
[...]
kubectl logs mariadb-78d7945894-zmg7k
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server latest started.
2023-11-30 13:31:58+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-11-30 13:31:58 0 [Note] Starting MariaDB 11.2.2-MariaDB-1:11.2.2+maria~ubu2204 source revision 929532a9426d085111c24c63de9c23cc54382259 as process 1
2023-11-30 13:31:58 0 [ERROR] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2023-11-30 13:32:28 0 [ERROR] mariadbd: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2023-11-30 13:32:28 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-11-30 13:32:28 0 [Note] InnoDB: Number of transaction pools: 1
2023-11-30 13:32:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-11-30 13:32:28 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-11-30 13:32:28 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-11-30 13:32:28 0 [Note] InnoDB: Completed initialization of buffer pool
2023-11-30 13:32:28 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2023-11-30 13:32:28 0 [Note] InnoDB: Check that you do not already have another mariadbd process using the same InnoDB data or log files.
2023-11-30 13:32:28 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-11-30 13:32:28 0 [Note] InnoDB: Starting shutdown...
2023-11-30 13:32:28 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-11-30 13:32:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-11-30 13:32:28 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-11-30 13:32:28 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'Aria'". Some plugins may be not loaded
2023-11-30 13:32:28 0 [ERROR] Failed to initialize plugins.
2023-11-30 13:32:28 0 [ERROR] Aborting
If you have any idea, I would be really thankful. If additional information is needed, please ask me!
You can't create multiple replicas of a MariaDB pod. MariaDB requires exclusive access to the storage directory; with your configuration, you're trying to share the storage directory between three instances of MariaDB. That's why things are crashing.
If you want multiple MariaDB instances, you would need to use a StatefulSet, giving each instance it's own storage volume, and then configure a galera cluster to enable replication between the instances.