I've a namespace I'm unable to delete in my Kubernetes cluster. When I run kubectl get ns traefik -o yaml
, I get the following:
apiVersion: v1
kind: Namespace
metadata:
annotations:
cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:28:59Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:29:00Z"}]}'
field.cattle.io/projectId: c-5g2hz:p-bl9sf
lifecycle.cattle.io/create.namespace-auth: "true"
creationTimestamp: "2021-06-11T20:28:58Z"
deletionTimestamp: "2021-07-04T07:21:20Z"
labels:
field.cattle.io/projectId: p-bl9sf
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:field.cattle.io/projectId: {}
f:labels:
.: {}
f:field.cattle.io/projectId: {}
f:status:
f:phase: {}
manager: agent
operation: Update
time: "2021-06-11T20:28:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cattle.io/status: {}
f:lifecycle.cattle.io/create.namespace-auth: {}
manager: rancher
operation: Update
time: "2021-06-11T20:28:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
.: {}
k:{"type":"NamespaceContentRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionContentFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionDiscoveryFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionGroupVersionParsingFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceFinalizersRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
manager: kube-controller-manager
operation: Update
time: "2021-07-04T07:21:26Z"
name: traefik
resourceVersion: "15400692"
uid: 4b198956-bbd5-4bdb-9dc6-9d53feda91e4
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2021-07-04T07:21:25Z"
message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
unable to handle the request'
reason: DiscoveryFailed
status: "True"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content successfully removed
reason: ContentRemoved
status: "False"
type: NamespaceContentRemaining
- lastTransitionTime: "2021-07-04T07:21:26Z"
message: All content-preserving finalizers finished
reason: ContentHasNoFinalizers
status: "False"
type: NamespaceFinalizersRemaining
phase: Terminating
And when I run kubectl delete ns traefik --v=10
, the last output is the following:
I0708 18:38:26.538676 31537 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/json" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" 'http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true'
I0708 18:38:27.013394 31537 round_trippers.go:445] GET http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true 200 OK in 474 milliseconds
I0708 18:38:27.013421 31537 round_trippers.go:451] Response Headers:
I0708 18:38:27.013427 31537 round_trippers.go:454] Access-Control-Allow-Origin: *
I0708 18:38:27.013450 31537 round_trippers.go:454] Date: Thu, 08 Jul 2021 16:38:27 GMT
I0708 18:38:27.013453 31537 round_trippers.go:454] Connection: keep-alive
I0708 18:38:27.013468 31537 request.go:708] Unexpected content type from the server: "": mime: no media type
I already tried to remove the finalizers as described on https://www.ibm.com/docs/en/cloud-private/3.2.0?topic=console-namespace-is-stuck-in-terminating-state but after some seconds I finally get EOF
:
> curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/traefik/finalize
EOF
Is anyone having any idea how I can delete the traefik namespace?
Posting this as a community wiki out of comments, feel free to edit and expand.
After analysing a state of the problematic namespace, this part was the main cause of the issue:
The issue was with
metric server
in kubernetes. Once themetric server
is available, namespace can be unblocked to be deleted.Similar issue was resolved in another answer on the stackoverflow.