K8s namespace terminating for 2 days

Question

I was attempting to remove fluxcd from a kubernetes cluster and it is stuck at deleting the namespace 'flux-system'. I've tried removing 'kubernetes' from the finalizers but getting a 404 when using curl to the api and 'invalid object' when trying '"/api/v1/namespaces/flux-system/finalize" -f flux-system.json'.

kubectl get ns flux-system -ojson | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/flux-system/finalize" -f -
Error from server (Conflict): Operation cannot be fulfilled on namespaces "flux-system": StorageError: invalid object, Code: 4, Key: /registry/namespaces/flux-system, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 6c59b2a0-d6b4-42de-bd2a-d148ad22ec19, UID in object meta:

curl -v -k -H "Content-type: application/json" -X PUT --data-binary @flux.json http://127.0.0.1:8001/k8s/clusters/c-5hqd5/v1/flux-system/finalize
 *   Trying 127.0.0.1...
 * TCP_NODELAY set
 * Connected to 127.0.0.1 (127.0.0.1) port 8001 (#0)
> PUT /k8s/clusters/c-5hqd5/v1/flux-system/finalize HTTP/1.1
> Host: 127.0.0.1:8001
> User-Agent: curl/7.64.1
> Accept: */*
> Content-type: application/json
> Content-Length: 3180
> Expect: 100-continue
>
 * Done waiting for 100-continue
 * We are completely uploaded and fine
< HTTP/1.1 100 Continue
< HTTP/1.1 404 Not Found
< Content-Length: 0
< Date: Fri, 27 May 2022 15:34:58 GMT
< Strict-Transport-Security: max-age=15724800; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Content-Type-Options: nosniff
<
Connection #0 to host 127.0.0.1 left intact
Closing connection 0

• Rancher v2.5.7
• Kubernets v1.20.11

kubectl edit namespace flux-system

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2022-05-20T17:33:54Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2022-05-20T17:33:54Z"}]}'
    field.cattle.io/projectId: c-5hqd5:p-4hbs6
    lifecycle.cattle.io/create.namespace-auth: "true"
  creationTimestamp: "2022-05-20T17:33:53Z"
  deletionTimestamp: "2022-05-20T21:46:36Z"
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/version: v0.28.5
    field.cattle.io/projectId: p-4hbs6
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
  name: flux-system
  resourceVersion: "147598092"
  uid: 6c59b2a0-d6b4-42de-bd2a-d148ad22ec19
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2022-05-20T21:46:41Z"
    message: 'Discovery failed for some groups, 2 failing: unable to retrieve the
      complete list of server APIs: flowcontrol.apiserver.k8s.io/v1beta1: the server
      could not find the requested resource, node.k8s.io/v1: the server could not
      find the requested resource'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2022-05-20T21:46:42Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2022-05-20T21:46:42Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2022-05-20T21:46:56Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2022-05-20T21:46:42Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

Answer 1

I read this as; you are attempting to apply an object with a UID that already exists, and the API is refusing that update, because there's a UID in the metadata.

Generally, re-applying objects read from the API via kubectl causes issues because they have previously passed though the admission controller mutating webhook; they have fields set that should be managed by the API, not via the CLI.

If you want to remove the finalizer on the namespace, my advice is to do it with kubectl edit, and see whether that allows you to remove them.

Answer 2

As i understand that you are trying to remove flux from the cluster the below command can be tried

flux uninstall --namespace=namespace