Senior DevOps / SRE Kubernetes Troubleshooting & Pod Lifecycle Core Troubleshooting

Q: How do you troubleshoot ImagePullBackOff?

Comprehensive diagnostic runbook for isolating the 4 primary root causes of ImagePullBackOff: image name/tag typos, missing or expired registry credentials, Docker Hub rate limiting (429), and VPC/DNS network egress failures.

#Kubernetes #ImagePullBackOff #Docker #ECR #ACR #imagePullSecrets

🎙️ Candidate Opening & Architectural Context

"ImagePullBackOff means kubelet tried to pull the container image from the registry, failed, and is backing off exponentially. My troubleshooting begins by running 'kubectl describe pod <pod-name>' and reading the exact container runtime error in the Events."

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Inspect Describe Events for the Exact Error String

Run kubectl describe pod <pod> -n <ns> and check the bottom Events section:

Error: ImagePullBackOff is preceded by Failed to pull image <image-name>: rpc error: code = NotFound / Unknown.
The exact error string immediately categorizes the failure into one of 4 root causes.

2️⃣

Root Cause 1: Image Name or Tag Typo / Non-Existent Image

Error: manifest unknown or repository does not exist:

Verify the repository URL, image name, and tag in spec.containers[0].image.
Check if the CI/CD pipeline actually pushed the image to ECR/ACR/DockerHub, or if the build job failed before the push stage.
Check for architecture mismatch (e.g. pushed ARM64 image while nodes are AMD64).

3️⃣

Root Cause 2: Authentication & Missing imagePullSecrets

Error: 401 Unauthorized or 403 Forbidden / Access Denied:

Private registries require Kubernetes credentials. Check if the Pod or its ServiceAccount references imagePullSecrets.
Check secret existence: kubectl get secret <secret-name> -o yaml.
In AWS EKS: Verify node IAM instance profile has AmazonEC2ContainerRegistryReadOnly or IRSA is configured.
In Azure AKS: Verify AKS kubelet identity has AcrPull role assignment on the Azure Container Registry (ACR).

4️⃣

Root Cause 3 & 4: Rate Limiting (429) & Network / Egress Blocks

Error: toomanyrequests: You have reached your pull rate limit or i/o timeout:

Docker Hub 429: Free tier limits anonymous pulls to 100 per 6 hours. Solution: Mirror images to private ECR/ACR or add authenticated Docker Hub pull secret.
Network / DNS Timeout: Worker nodes in private subnets cannot reach external registries if NAT Gateway is down, security group blocks outbound 443, or CoreDNS fails to resolve registry domain.
Quick Test from Node: SSH into worker node or run a debug pod: crictl pull <image-name> to test pull directly.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Read the exact error in 'kubectl describe pod': 'manifest unknown' = image tag typo or unpushed image; '401/403' = missing imagePullSecret or IAM/AcrPull role; '429' = Docker Hub rate limit; 'i/o timeout' = NAT Gateway or egress firewall block."

⚡ 60-Second Elevator Pitch Talking Points

Run 'kubectl describe pod <name>' and examine the Events message.
Case 1: 'manifest unknown' -> Image tag typo or CI/CD failed to push image.
Case 2: '401 Unauthorized' -> Missing imagePullSecret in pod spec or node lacks ECR/ACR IAM pull permissions.
Case 3: '429 Too Many Requests' -> Docker Hub rate limit; switch to private registry mirror (ECR/ACR).
Case 4: 'Connection timeout' -> Node in private subnet has no egress route to NAT Gateway or Security Group blocks 443 outbound.