Q: An engineer deploys an image to production and it immediately enters 'ImagePullBackOff'. The developer insists 'it runs fine on my local machine'. Walk me through your diagnostic tree to isolate the root cause across: tag typos, expired registry credentials, VPC NAT egress failure, and ARM64 vs AMD64 architecture mismatch.

Inspect the pod event log to classify the failure into one of the 4 primary root cause categories:

• Describe Pod Events: Run kubectl describe pod <pod-name> | grep -A 10 Events:.
• Identify the RPC Status: Look for specific error strings:
  • 404 Not Found / manifest unknown: Typo in image name or tag.
  • 401 Unauthorized / 403 Forbidden: Missing or expired registry credentials.
  • 429 Too Many Requests: Docker Hub anonymous pull rate limit exceeded (100 pulls per 6 hours).
  • i/o timeout / dial tcp: Node network routing or VPC egress failure.
  • no match for platform in manifest: Multi-architecture mismatch (e.g. ARM64 vs AMD64).

For private registries, verify how Kubelet authenticates with the remote registry:

• Inspect imagePullSecrets in Pod Spec: Verify the pod defines imagePullSecrets: [{name: <secret-name>}] or that the ServiceAccount defines it.
• Inspect Secret Contents: Run kubectl get secret <secret-name> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d. Verify the registry URL, auth token, and expiration timestamp.
• AWS ECR 12-Hour Token Expiration: ECR authorization tokens expire every 12 hours. If using standard Kubernetes secrets rather than AWS IAM node roles, the refresh cron job has likely stalled.
• AWS IAM Node Role / EKS Pod Identity: On Amazon EKS, ensure the worker node IAM instance profile contains the managed policy AmazonEC2ContainerRegistryReadOnly.

Developers building images on Apple Silicon (M1/M2/M3) frequently push ARM64 images that x86 worker nodes cannot execute:

• Check Error Message: failed to extract layer: no match for platform in manifest: not found or exec format error.
• Inspect Image Architecture via Skopeo / Docker: Run docker manifest inspect <image-name>:<tag> to check the architecture field.
• Multi-Arch Build Fix: Build using Docker Buildx for multiple platforms in CI/CD:

  docker buildx build --platform linux/amd64,linux/arm64 -t <image>:<tag> --push .

If the error is an I/O timeout, verify that worker nodes in private subnets have outbound routing to the registry:

• Check Route Tables & NAT Gateway: Confirm private subnet route tables have 0.0.0.0/0 pointed to an active AWS NAT Gateway.
• Verify VPC Endpoints for ECR: In strict zero-internet private clusters, confirm both VPC Interface Endpoints are deployed: com.amazonaws.region.ecr.api and com.amazonaws.region.ecr.dkr.
• Verify S3 Gateway Endpoint: AWS ECR stores image layers in Amazon S3 buckets. If you have ECR VPC endpoints but NO Amazon S3 Gateway Endpoint (com.amazonaws.region.s3), manifest downloads will succeed, but actual layer downloads will hang and time out!

Isolate whether the failure is cluster-wide or specific to a single worker node by testing CRI pulls directly on the host:

• SSH to Worker Node: Run: crictl pull <full-image-name>:<tag>.
• Observe Terminal Feedback: This provides instantaneous debugging output, bypassing the 5-minute Kubernetes backoff timer.
• Check containerd Registry Mirrors: Verify /etc/containerd/config.toml to ensure registry mirror endpoints are reachable.

Resolve the issue and observe Kubelet successfully launching the pod:

• Force Immediate Pull: Delete the stuck pod (kubectl delete pod <pod-name>) to reset the backoff counter.
• Confirm Running Status: Confirm STATUS: Running and READY: 1/1.
• Lab Runbook: Practice fixing ImagePullBackOff, ECR auth, and VPC endpoints in DevOps Lab: ImagePullBackOff Hands-On Runbook ↗.