⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 96 of 207 in Kubernetes
Staff SRE / Principal Architect [L3] Kubernetes Advanced Scenarios Staff SRE Scenario [L3]
🎯 Target Role / Context: Kubernetes Scenario #86 (L3) · Advanced Scenarios

Q: Your admission webhook is blocking all pod creation cluster-wide. How do you recover?

If webhook server is down, set failurePolicy: Ignore on the webhook or delete the MutatingWebhookConfiguration object.

#Kubernetes #Advanced Scenarios #L3 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>When an admission webhook is blocking all pod creation, recover by removing or disabling the failing webhook configuration (or fixing its backend), then restoring it with a safe <code>failurePolicy</code>.</strong> This happens when a webhook with <code>failurePolicy: Fail</code> points at a service that is down, has an expired certificate, or depends on pods that cannot start because of the webhook itself, a circular dependency. Interviewers want to see calm triage: confirm the symptom from the error text, identify the webhook, break the deadlock using <code>kubectl</code> access to the API server, and add guardrails so it cannot recur."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>When an admission webhook is blocking all pod creation, recover by removing or disabling the failing webhook configuration (or fixing its backend), then restoring it with a safe <code>failurePolicy</code>.</strong> This happens when a webhook with <code>failurePolicy: Fail</code> points at a service that is down, has an expired certificate, or depends on pods that cannot start because of the webhook itself, a circular dependency. Interviewers want to see calm triage: confirm the symptom from the error text, identify the webhook, break the deadlock using <code>kubectl</code> access to the API server, and add guardrails so it cannot recur.

1️⃣

Recognise a webhook outage from the error

The API server tells you exactly which webhook rejected the request, so read the error before changing anything.

  • Typical message: Internal error occurred: failed calling webhook "validate.example.com": Post "https://svc.ns.svc:443/validate?timeout=10s": no endpoints available. Other variants mention x509: certificate has expired or context deadline exceeded.
  • Scope: ReplicaSet events show FailedCreate with that text, so Deployments appear healthy but have zero new pods. Existing running pods are unaffected.
  • Mutating vs validating: both can block. Mutating webhooks run first, so check both configuration types.
  • Who is affected: only requests matching the webhook rules (resources, operations, namespace and object selectors). A webhook matching pods CREATE in all namespaces breaks everything.
  • Quick triage clue: if only some namespaces fail, look at namespaceSelector and objectSelector first. Cluster-wide failure points at a rule with no selector, which is the scenario in this question.
2️⃣

Hands-on: admission webhook blocking all pod creation, find and inspect

List the webhook configurations and read the rules, failurePolicy and backing Service.

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

kubectl get validatingwebhookconfiguration my-policy -o yaml | \
  grep -E 'name:|failurePolicy|timeoutSeconds|namespace:|service:' 

# Is the backend alive?
kubectl -n policy get endpoints my-policy-webhook
kubectl -n policy get pods -l app=my-policy-webhook
  • List: both validatingwebhookconfigurations and mutatingwebhookconfigurations are cluster-scoped.
  • Match the name: use the webhook name from the error to find the owning configuration, and read its clientConfig.service.
  • Check the backend: confirm whether the Service has endpoints and whether the backing pods are running.
  • Look at policy: failurePolicy: Fail plus broad rules is the dangerous combination, while Ignore would have let requests through.
  • Timing and audit: kubectl get events -A --field-selector reason=FailedCreate shows how many controllers are affected and since when, which helps estimate the incident window and the number of objects admitted during any bypass.
Advertisement
3️⃣

Recover: bypass the webhook safely

Prefer the least destructive option that restores pod creation, and save the original configuration first.

# Save the config first
kubectl get validatingwebhookconfiguration my-policy -o yaml > my-policy.yaml

# Option B: fail open for the first webhook entry
kubectl patch validatingwebhookconfiguration my-policy --type=json \
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

# Option C: remove it entirely (restore later with kubectl apply -f my-policy.yaml)
kubectl delete validatingwebhookconfiguration my-policy
  • Back up: kubectl get validatingwebhookconfiguration my-policy -o yaml > my-policy.yaml so you can restore it exactly.
  • Option A, fix the backend: if it is a plain outage, restart or scale the webhook Deployment, renew the certificate, or fix the NetworkPolicy. This is best when it works quickly.
  • Option B, relax policy: patch failurePolicy to Ignore so requests pass while the backend is down. Note that this also drops enforcement.
  • Option C, delete: deleting the configuration is the last resort when the webhook's own pods cannot start. Controllers will recreate stuck ReplicaSet pods once admission succeeds.
  • Order of operations: disable or relax the webhook that is down, not every webhook. Other healthy webhooks keep enforcing policy, so avoid deleting all configurations in panic.
Pro Tip: If you cannot reach the API server at all, the problem is not an admission webhook. Webhooks only block write requests that match their rules, so get and delete on the webhook configuration objects themselves are normally never intercepted.
4️⃣

Verify and restore enforcement

Confirm pods can be created, then bring the webhook back in a controlled way.

  • Smoke test: kubectl run test --image=registry.k8s.io/pause:3.9 --dry-run=server exercises admission without creating anything.
  • Retry stuck workloads: ReplicaSets retry with backoff. A kubectl rollout restart speeds recovery for deployments that are stuck.
  • Re-enable: restore the saved configuration only after the backend has ready endpoints and a valid certificate, ideally in a staging namespace first.
  • Audit: review what was admitted without policy during the bypass window, using audit logs, since unvalidated objects may have slipped through.
  • Communicate: announce the temporary loss of enforcement, record the exact commands used, and set a ticket to review admitted objects and to restore the original configuration after the backend is healthy.
5️⃣

Prevent a repeat

Webhook design determines the blast radius. Most outages come from the same few configuration mistakes.

  • Exclude system namespaces: use namespaceSelector to skip kube-system and the webhook's own namespace, so it can never block its own pods or critical components.
  • Short timeouts: set timeoutSeconds low (default is 10 seconds, maximum 30) so a hung backend does not stall every request.
  • Choose the failure mode deliberately: use Fail only for real security controls, and run the backend with several replicas, a PodDisruptionBudget and anti-affinity.
  • Certificate automation: manage the serving certificate with cert-manager or the controller's rotation, with an alert on expiry, and alert on webhook latency and error metrics from the API server.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Back up the webhook configuration, switch failurePolicy to Ignore or delete it to restore admission, then re-enable it only with system namespaces excluded."
⚡ 60-Second Elevator Pitch Talking Points
  • A webhook with failurePolicy Fail and a dead backend rejects every matching request, so no pods can be created even though running pods are fine.
  • I read the error for the webhook name, inspect its rules and Service endpoints, back it up, then fix the backend or patch it to Ignore or delete it, and prove recovery with a server-side dry run.
  • Afterward I exclude kube-system and its own namespace, set short timeouts, run replicas with a PDB, and automate certificate rotation.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Deleting the webhook configuration without saving it You lose the exact rules and CA bundle, making a faithful restore difficult.
  2. Letting the webhook match its own namespace If its pods are down, they cannot be recreated, which produces a circular deadlock.
  3. Using failurePolicy Fail for non-critical checks A routine outage of the policy service then blocks the whole cluster.
  4. Forgetting to audit after a bypass Objects created while enforcement was off may violate policy and go unnoticed.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →