Namespace Issues when Removing CRD/Operators
source link: https://www.tuicool.com/articles/hit/Bzy22a7
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
With the latest release of Pure Service Orchestrator, we added support for a non-Helm installation for environments that do not allow Helm. This new method uses an Operator to setup and install PSO. The result is the same exact functionality but uses a security model more agreeable to some K8s distro vendors.
I do live demos of PSO a handful of times a day. Even though I use Terraform and Ansible to automate the creation of my lab K8s clusters I don’t want to do this many times a day. I usually just tear down PSO and leave my cluster ready for the next demo.
Removing the CRD and the Namespace created when installing the Operator has a couple of issues. One small issue is the Operator method creates a new namespace “pso-operator”. This is the default name, and you can choose your own namespace name during install time. I often choose “pso” for simplicity. As we have discovered, deleting a namespace that had a CRD installed into hangs in the status “Terminating”, for like, forever. FOR-EV-ER. This seems to be an issue dating back quite a ways in K8s land.
https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-448120772
From a couple of GitHub issues and the help of Simon “I don’t do the twitter” Dodsley This is the process for deleting the CRD first and the Namespace. This method keeps the namespace form hanging in the state “Terminating”.
# Removing the pso-operator kubectl delete all --all -n pso-operator # If you haven't don't it already don't delete the namespace yet. kubectl get ns NAME STATUS AGE default Active 2d21h kube-public Active 2d21h kube-system Active 2d21h pso-operator Active 14h kubectl get crd NAME CREATED AT psoplugins.purestorage.com 2019-04-17T01:37:31Z # ok so... kubectl delete crd psoplugins.purestorage.com customresourcedefinition.apiextensions.k8s.io "psoplugins.purestorage.com" deleted # does it hang? yeah it does ^C # stuck terminating? kubectl describe crd psoplugins.purestorage.com # snipping non-relevant output ... Conditions: Last Transition Time: 2019-04-17T01:37:31Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: <nil> Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established Last Transition Time: 2019-04-18T13:54:36Z Message: CustomResource deletion is in progress Reason: InstanceDeletionInProgress Status: True Type: Terminating Stored Versions: v1 # Run this command to allow it to delete kubectl patch crd/psoplugins.purestorage.com -p '{"metadata":{"finalizers":[]}}' --type=merge customresourcedefinition.apiextensions.k8s.io/psoplugins.purestorage.com patched # Re-run the crd delete kubectl delete crd psoplugins.purestorage.com # Confirm it is gone kubectl get crd No resources found. # Remove the Namespace kubectl delete ns pso-operator namespace "pso-operator" deleted #Verify removal kubectl get ns NAME STATUS AGE default Active 2d21h kube-public Active 2d21h kube-system Active 2d21h
If you sort of ignored my warning above and tried to remove the namespace BEFORE successfully removing the CRD follow the following procedure.
Namespace Removal
# Find that pesky 'Terminating' namespace kubectl get ns NAME STATUS AGE default Active 2d20h kube-public Active 2d20h kube-system Active 2d20h pso Active 13h pso-operator Terminating 35h kubectl cluster-info # run the kube-proxy kubectl proxy & # output the namespace to json kubectl get namespace pso-operator -o json >tmp.json # Edit the tmp.json to remove the finalizer the spec: should look like this: "spec": { "finalizers": [ ] }, # Now send that tmp.json to the API server curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/pso-operator/finalize # Check your namespaces kubectl get ns NAME STATUS AGE default Active 2d20h kube-public Active 2d20h kube-system Active 2d20h pso Active 13h # disable the kube-proxy, bring it back to the foreground and ctrl-C fg ^C
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK