Q: Someone accidentally ran `kubectl delete clusterrolebinding cluster-admin` and deleted the cluster admin binding. Now no one can manage the cluster. What do you do?
This is a serious situation. If you're locked out of the API server entirely:
#Kubernetes #RBAC & Security #L3 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
""When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This is a serious situation. If you're locked out of the API server entirely:
- SSH directly to a control plane node.
- Use
kubectlwith the admin kubeconfig at/etc/kubernetes/admin.conf(setKUBECONFIG=/etc/kubernetes/admin.conf). This uses certificate-based auth that bypasses RBAC. - Recreate the cluster-admin binding:
2️⃣
Remediation & Permanent Safeguards
Prevention: Never give a single ClusterRoleBinding a name that might be confused with a default. Back up RBAC configs. Use --dry-run=client before destructive commands. --- ## 🔵 Scaling & Performance
kubectl create clusterrolebinding cluster-admin \
--clusterrole=cluster-admin \
--user=<your-user>
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: SSH directly to a control plane node.."
⚡ 60-Second Elevator Pitch Talking Points
- SSH directly to a control plane node.
- Use kubectl with the admin kubeconfig at /etc/kubernetes/admin.conf (set KUBECONFIG=/etc/kubernet...
- Recreate the cluster-admin binding:
Advertisement