kubernetes python API - fix faulty namespace

Question

I have an issue with some namespaces stuck in Terminating state.

the problem is with the finalizer and as soon as it's cleaned up (replace ['kubernetes'] with []) it terminates.

I clear them up manually with a bit of a hack:

kubectl get ns <namespace> -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/auth/finalize" -f -

What I want to do is a simple python script to detect all stuck namespaces and "fix" them one by one.

from kubernetes import client as cli
from kubernetes import config as conf
import kubernetes  # to make type matching more complete

conf.load_kube_config()
client = cli.CoreV1Api()
nss = client.list_namespace()

# Find dying namespaces:
dying = []
for ns in nss.items:
    if ns.status.phase == "Terminating":
        print(f"------ {ns.metadata.name} is dying {ns.status.phase}")
        dying.append(ns)

print("dyingspaces: ", len(dying))


# Try to clean up the bad guys:
for ns in dying:
    # make sure:
    if ns.status.phase == "Terminating":
        name = ns.metadata.name
        ns.spec.finalizers = []
        # apply edited object:
        rep = client.replace_namespace(name, ns)

There's no change after executing the code, no exceptions thrown and the return looks like the same json including the original finalizer.
My guess is the python call doesn't work the same way or the --raw flag doesn't exist in the python API or I need an entirely different function. I tried to use the patch_namespace with the same arguments to no avail.

Answer 1

The concern you have may be because of the Python script to fix terminating namespaces due to a combination factors:

1. Patching vs Replacing: While your script attempts to modify finalizers field using replace\_namespace, this method replaces the entire namespace object. You need to use patch\_namespace instead, which allows specific fields to be updated.

2. Strategic Merge Patch: Kubernetes uses strategic merge patching by default, which interprets the path document differently than a regular JSON patch. You need to specify the strategicMergePatch type in your patch request for the finalizers field update to work.

3. Accessing Raw Data: The --raw flag is specific to the kubectl command-line tool and doesn't translate directly to the Python API. However, you can achieve the same effect by constructing a raw patch document using a Python dictionary with the desired field changes.

You may refer to this sample as reference:

from kubernetes import client as cli
from kubernetes import config as conf
import json

conf.load_kube_config()
client = cli.CoreV1Api()
nss = client.list_namespace()

# Find dying namespaces:
dying = []
for ns in nss.items:
    if ns.status.phase == "Terminating":
        print(f"------ {ns.metadata.name} is dying {ns.status.phase}")
        dying.append(ns)

print("dying namespaces:", len(dying))

# Try to clean up the bad guys:
for ns in dying:
    # make sure:
    if ns.status.phase == "Terminating":
        name = ns.metadata.name
        patch = {"spec": {"finalizers": []}, "type": "StrategicMergePatch"}
        # apply patch:
        client.patch_namespace(name, patch)
        print(f"Patched {name} to remove finalizers")

Remember to restart any pods or services affected by the terminating namespaces after successfully removing the finalizers. Additionally, this approach only removes the kubernetes finalizer, so make sure to investigate if other finalizers are present and need to be addressed individually.