Kubernetes tips #2 – force-remove namespace

By | 2021-02-17
0 Comment

When a namespace stucks in “terminating” state you have manually solve the blocking conditions.

At first check the namespace:

kubectl edit namespace <YOUR ROGUE NAMESPACE>

When the monitoring or metrics is broken there is similar message:

status:
  conditions:
  - lastTransitionTime: "<DATE>"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
      complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
      unable to handle the request'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure

Check the metrics.k8s.io apiservice:

kubectl get apiservice | grep metrics.k8s.io

If there are missing endpoints or other error which caused the “AVAILABLE” field set to False, you need to deal it. Easiest way to remove it.

kubectl get apiservice v1beta1.metrics.k8s.io
NAME                     SERVICE                         AVAILABLE                  AGE
v1beta1.metrics.k8s.io   monitoring/prometheus-adapter   False (MissingEndpoints)   130d

kubectl delete apiservice v1beta1.metrics.k8s.io
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted

You have to wait a little time and the terminated namespace will be removed.

If it is not removed automatically, you have to force-remove it. Usually it waits only for the “finalizers”.

You can edit manually the resource and remove the finalizer section.

kubectl edit namespace <YOUR ROGUE NAMESPACE>

Or you can do it with command line using kubectl, jq and curl.

(
NAMESPACE=<YOUR ROGUE NAMESPACE>
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
)

It starts the proxy at 128.0.01:8001 by default, fetch the namespace definition and empties the finalizers section with jq. Then it puts the modified definition via the api proxy.

Original source of second part: https://stackoverflow.com/questions/52369247/namespace-stuck-as-terminating-how-do-i-remove-it