Can't delete the namespace which is the MYSQL InnoDB Cluster installed in kubernetes

Question

I installed a MYSQL Operator and a MYSQL InnoDB Cluster in deferent namespaces. Then I was about to delete both. It was successful for the MYSQL-Operator to delete. But I failed to delete the InnoDB Cluster. So I deleted each of the InnoDB Cluster components with force option. Some was successful but some was failed. So I tried to delete the namespace ifself with the last thing I could think of. But that ended up failure. It would be great that someone help me solve this problem. (Please let me know if you need more information.)

$ kubectl get namespace mycluster
mycluster Terminating 5d1h

$ kubectl get all -n mycluster
No resources found in mycluster namespace

$ kubectl get namespace mycluster
apiVersion: v1 kind: Namespace metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"creationTimestamp":null,"name":"mycluster"},"spec":{},"status":{}} creationTimestamp: "2024-01-13T01:14:56Z" deletionTimestamp: "2024-01-13T02:30:18Z" labels: kubernetes.io/metadata.name: mycluster name: mycluster resourceVersion: "5011759" uid: a24383af-79b3-4e04-8ec0-fc049431d936 spec: finalizers:

• kubernetes status: conditions:
• lastTransitionTime: "2024-01-13T02:30:25Z" message: All resources successfully discovered reason: ResourcesDiscovered status: "False" type: NamespaceDeletionDiscoveryFailure
• lastTransitionTime: "2024-01-13T02:30:25Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure
• lastTransitionTime: "2024-01-13T02:30:25Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure
• lastTransitionTime: "2024-01-13T02:30:25Z" message: 'Some resources are remaining: innodbclusters.mysql.oracle.com has 1 resource instances' reason: SomeResourcesRemain status: "True" type: NamespaceContentRemaining
• lastTransitionTime: "2024-01-13T02:30:25Z" message: 'Some content in the namespace has finalizers remaining: mysql.oracle.com/cluster in 1 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining phase: Terminating

$ kubectl describe namespace mycluster
Name: argocd Labels: kubernetes.io/metadata.name=argocd Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: cert-manager Labels: kubernetes.io/metadata.name=cert-manager Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: db-cluster Labels: kubernetes.io/metadata.name=db-cluster Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: default Labels: kubernetes.io/metadata.name=default Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: dev Labels: kubernetes.io/metadata.name=dev Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: ingress-nginx Labels: kubernetes.io/metadata.name=ingress-nginx Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: kube-node-lease Labels: kubernetes.io/metadata.name=kube-node-lease Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: kube-public Labels: kubernetes.io/metadata.name=kube-public Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: kube-system Labels: kubernetes.io/metadata.name=kube-system Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: mycluster Labels: kubernetes.io/metadata.name=mycluster Annotations: <none> Status: Terminating Conditions: Type Status LastTransitionTime Reason Message

NamespaceDeletionDiscoveryFailure False Sat, 13 Jan 2024 11:30:25 +0900 ResourcesDiscovered All resources successfully discovered NamespaceDeletionGroupVersionParsingFailure False Sat, 13 Jan 2024 11:30:25 +0900 ParsedGroupVersions All legacy kube types successfully parsed NamespaceDeletionContentFailure False Sat, 13 Jan 2024 11:30:25 +0900 ContentDeleted All content successfully deleted, may be waiting on finalization NamespaceContentRemaining True Sat, 13 Jan 2024 11:30:25 +0900 SomeResourcesRemain Some resources are remaining: innodbclusters.mysql.oracle.com has 1 resource instances NamespaceFinalizersRemaining True Sat, 13 Jan 2024 11:30:25 +0900 SomeFinalizersRemain Some content in the namespace has finalizers remaining: mysql.oracle.com/cluster in 1 resource instances

No resource quota.

No LimitRange resource.

Name: mysql-operator Labels: kubernetes.io/metadata.name=mysql-operator Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

Name: redis Labels: kubernetes.io/metadata.name=redis Annotations: <none> Status: Active

No resource quota.

No LimitRange resource.

I tried to delete the namespace in many ways.

Answer 1

Solved.

kubectl patch crd mysqlbackups.mysql.oracle.com -p '{"metadata":{"finalizers": []}}' --type=merge

I patched crd's finalizers.

Answer 2

I am also facing the same issue but the above solution didn't work for me.

Labels:       kubernetes.io/metadata.name=mysql-operator
Annotations:  <none>
Status:       Terminating
Conditions:
  Type                                         Status  LastTransitionTime               Reason                Message
  ----                                         ------  ------------------               ------                -------
  NamespaceDeletionDiscoveryFailure            False   Mon, 29 Jan 2024 14:12:46 +0530  ResourcesDiscovered   All resources successfully discovered
  NamespaceDeletionGroupVersionParsingFailure  False   Mon, 29 Jan 2024 14:12:46 +0530  ParsedGroupVersions   All legacy kube types successfully parsed
  NamespaceDeletionContentFailure              False   Mon, 29 Jan 2024 14:12:46 +0530  ContentDeleted        All content successfully deleted, may be waiting on finalization
  NamespaceContentRemaining                    True    Mon, 29 Jan 2024 14:12:46 +0530  SomeResourcesRemain   Some resources are remaining: innodbclusters.mysql.oracle.com has 1 resource instances
  NamespaceFinalizersRemaining                 True    Mon, 29 Jan 2024 14:12:46 +0530  SomeFinalizersRemain  Some content in the namespace has finalizers remaining: mysql.oracle.com/cluster in 1 resource instances

arun@ArunLAL555:/mnt/f/mysql-operator/innodb-cluster$ kubectl patch crd mysqlbackups.mysql.oracle.com -p '{"metadata":{"finalizers": []}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/mysqlbackups.mysql.oracle.com patched (no change)
arun@ArunLAL555:/mnt/f/mysql-operator/innodb-cluster$ kubectl patch crd kopfpeerings.zalando.org -p '{"
metadata":{"finalizers": []}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/kopfpeerings.zalando.org patched (no change)
arun@ArunLAL555:/mnt/f/mysql-operator/innodb-cluster$ kubectl patch crd clusterkopfpeerings.zalando.org -p '{"metadata":{"finalizers": []}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/clusterkopfpeerings.zalando.org patched (no change)
arun@ArunLAL555:/mnt/f/mysql-operator/innodb-cluster$ k get ns
NAME                  STATUS        AGE
actions-runner        Active        6d2h
mysql-operator        Terminating   4d8h
pet-clinic-app        Active        18d
pet-clinic-db         Active        19d
prometheus-operator   Active        37d
prometheus-thanos     Active        27d
stateful-workloads    Active        9d
thanos                Active        27d

The above solution works, you can try the following removing finalizers method to remove the stuck Namespaces on the Terminating state

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.metadata.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

This will automatically identify the namespaces which are in the terminating state and manipulate the finalizer to empty them.