Q: Your admission webhook is blocking all pod creation cluster-wide. How do you recover?
If webhook server is down, set failurePolicy: Ignore on the webhook or delete the MutatingWebhookConfiguration object.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>When an admission webhook is blocking all pod creation, recover by removing or disabling the failing webhook configuration (or fixing its backend), then restoring it with a safe <code>failurePolicy</code>.</strong> This happens when a webhook with <code>failurePolicy: Fail</code> points at a service that is down, has an expired certificate, or depends on pods that cannot start because of the webhook itself, a circular dependency. Interviewers want to see calm triage: confirm the symptom from the error text, identify the webhook, break the deadlock using <code>kubectl</code> access to the API server, and add guardrails so it cannot recur.
Recognise a webhook outage from the error
The API server tells you exactly which webhook rejected the request, so read the error before changing anything.
- Typical message:
Internal error occurred: failed calling webhook "validate.example.com": Post "https://svc.ns.svc:443/validate?timeout=10s": no endpoints available. Other variants mentionx509: certificate has expiredorcontext deadline exceeded. - Scope: ReplicaSet events show
FailedCreatewith that text, so Deployments appear healthy but have zero new pods. Existing running pods are unaffected. - Mutating vs validating: both can block. Mutating webhooks run first, so check both configuration types.
- Who is affected: only requests matching the webhook rules (resources, operations, namespace and object selectors). A webhook matching
podsCREATE in all namespaces breaks everything. - Quick triage clue: if only some namespaces fail, look at
namespaceSelectorandobjectSelectorfirst. Cluster-wide failure points at a rule with no selector, which is the scenario in this question.
Hands-on: admission webhook blocking all pod creation, find and inspect
List the webhook configurations and read the rules, failurePolicy and backing Service.
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfiguration my-policy -o yaml | \
grep -E 'name:|failurePolicy|timeoutSeconds|namespace:|service:'
# Is the backend alive?
kubectl -n policy get endpoints my-policy-webhook
kubectl -n policy get pods -l app=my-policy-webhook
- List: both
validatingwebhookconfigurationsandmutatingwebhookconfigurationsare cluster-scoped. - Match the name: use the webhook name from the error to find the owning configuration, and read its
clientConfig.service. - Check the backend: confirm whether the Service has endpoints and whether the backing pods are running.
- Look at policy:
failurePolicy: Failplus broad rules is the dangerous combination, whileIgnorewould have let requests through. - Timing and audit:
kubectl get events -A --field-selector reason=FailedCreateshows how many controllers are affected and since when, which helps estimate the incident window and the number of objects admitted during any bypass.
Recover: bypass the webhook safely
Prefer the least destructive option that restores pod creation, and save the original configuration first.
# Save the config first
kubectl get validatingwebhookconfiguration my-policy -o yaml > my-policy.yaml
# Option B: fail open for the first webhook entry
kubectl patch validatingwebhookconfiguration my-policy --type=json \
-p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'
# Option C: remove it entirely (restore later with kubectl apply -f my-policy.yaml)
kubectl delete validatingwebhookconfiguration my-policy
- Back up:
kubectl get validatingwebhookconfiguration my-policy -o yaml > my-policy.yamlso you can restore it exactly. - Option A, fix the backend: if it is a plain outage, restart or scale the webhook Deployment, renew the certificate, or fix the NetworkPolicy. This is best when it works quickly.
- Option B, relax policy: patch
failurePolicytoIgnoreso requests pass while the backend is down. Note that this also drops enforcement. - Option C, delete: deleting the configuration is the last resort when the webhook's own pods cannot start. Controllers will recreate stuck ReplicaSet pods once admission succeeds.
- Order of operations: disable or relax the webhook that is down, not every webhook. Other healthy webhooks keep enforcing policy, so avoid deleting all configurations in panic.
Verify and restore enforcement
Confirm pods can be created, then bring the webhook back in a controlled way.
- Smoke test:
kubectl run test --image=registry.k8s.io/pause:3.9 --dry-run=serverexercises admission without creating anything. - Retry stuck workloads: ReplicaSets retry with backoff. A
kubectl rollout restartspeeds recovery for deployments that are stuck. - Re-enable: restore the saved configuration only after the backend has ready endpoints and a valid certificate, ideally in a staging namespace first.
- Audit: review what was admitted without policy during the bypass window, using audit logs, since unvalidated objects may have slipped through.
- Communicate: announce the temporary loss of enforcement, record the exact commands used, and set a ticket to review admitted objects and to restore the original configuration after the backend is healthy.
Prevent a repeat
Webhook design determines the blast radius. Most outages come from the same few configuration mistakes.
- Exclude system namespaces: use
namespaceSelectorto skipkube-systemand the webhook's own namespace, so it can never block its own pods or critical components. - Short timeouts: set
timeoutSecondslow (default is 10 seconds, maximum 30) so a hung backend does not stall every request. - Choose the failure mode deliberately: use
Failonly for real security controls, and run the backend with several replicas, a PodDisruptionBudget and anti-affinity. - Certificate automation: manage the serving certificate with cert-manager or the controller's rotation, with an alert on expiry, and alert on webhook latency and error metrics from the API server.
- A webhook with failurePolicy Fail and a dead backend rejects every matching request, so no pods can be created even though running pods are fine.
- I read the error for the webhook name, inspect its rules and Service endpoints, back it up, then fix the backend or patch it to Ignore or delete it, and prove recovery with a server-side dry run.
- Afterward I exclude kube-system and its own namespace, set short timeouts, run replicas with a PDB, and automate certificate rotation.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Deleting the webhook configuration without saving it You lose the exact rules and CA bundle, making a faithful restore difficult.
- Letting the webhook match its own namespace If its pods are down, they cannot be recreated, which produces a circular deadlock.
- Using failurePolicy Fail for non-critical checks A routine outage of the policy service then blocks the whole cluster.
- Forgetting to audit after a bypass Objects created while enforcement was off may violate policy and go unnoticed.