Q: A pod has been running fine for weeks and suddenly starts failing with `ImagePullBackOff`. Nothing in the pod spec changed. What could cause this?
Since nothing changed in the spec, suspect external changes:
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Kubernetes is a declarative desired state system; understanding the reconciliation loop is how you diagnose this quickly. The interviewer is testing: Image registry auth and image availability awareness.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Since nothing changed in the spec, suspect external changes:
- Registry credentials expired — imagePullSecret token rotated or expired.
- Image was deleted from the registry — someone deleted the tag from Docker Hub or ECR.
- Registry is down or unreachable — network issue or registry outage.
Remediation & Permanent Safeguards
Check: kubectl describe pod — the event will say exactly which registry returned what error (401 Unauthorized, 404 Not Found, etc.).
- Rate limiting — Docker Hub has pull rate limits for unauthenticated/free accounts.
- Private registry changed auth — ECR tokens expire every 12 hours if not refreshed.
- Registry credentials expired — imagePullSecret token rotated or expired.
- Image was deleted from the registry — someone deleted the tag from Docker Hub or ECR.
- Registry is down or unreachable — network issue or registry outage.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when configuring running fine weeks Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes when configuring running fine weeks
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending when configuring running fine weeks
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods when configuring running fine weeks Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.