⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 24 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Troubleshooting & Debugging Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #10 (L2) · Troubleshooting & Debugging

Q: Your init container is stuck and the main container never starts. How do you debug?

Debug a stuck Kubernetes init container by inspecting pod events with describe, checking init container logs, and verifying blocking service dependencies.

#Kubernetes #Troubleshooting & Debugging #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"When an init container is stuck, the Kubernetes kubelet halts pod initialization because init containers must complete sequentially and exit with code 0 before app containers start. The pod status remains frozen in <code>Init:0/1</code>, <code>Init:CrashLoopBackOff</code>, or <code>PodInitializing</code>. To diagnose the failure quickly, senior engineers follow a structured triage path: check pod status and termination exit codes, extract init container logs, test network dependencies, and resolve deadlocks."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When an init container is stuck, the Kubernetes kubelet halts pod initialization because init containers must complete sequentially and exit with code 0 before app containers start. The pod status remains frozen in <code>Init:0/1</code>, <code>Init:CrashLoopBackOff</code>, or <code>PodInitializing</code>. To diagnose the failure quickly, senior engineers follow a structured triage path: check pod status and termination exit codes, extract init container logs, test network dependencies, and resolve deadlocks.

1️⃣

Triage Why Your Init Container is Stuck Using Kubectl Logs

Init containers execute sequentially in the exact order declared under spec.initContainers. Understanding status codes is critical for triage.

  • Init:N/M status: In a pod with M init containers, N have successfully completed with exit code 0. If stuck at Init:0/1, the first init container is either hanging or actively running.
  • Init:CrashLoopBackOff: The init container completed with a non-zero exit code. Kubelet restarts it with an exponential backoff delay (10s, 20s, up to 5m).
  • Main container dormancy: The primary application containers in spec.containers remain in Waiting state with reason PodInitializing until all init containers succeed.
2️⃣

Step 1: Inspect pod status, container exit codes, and recent events

Execute kubectl describe to extract the init container's current state, last termination reason, exit code, and cluster events.

# Identify pods stuck in Init state
kubectl get pods -l app=payment-api

# Detailed inspection of container state and events
kubectl describe pod payment-api-7b89df5c4-x8k9l

# Extract exact state and exit codes via jsonpath
kubectl get pod payment-api-7b89df5c4-x8k9l -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'
  • State inspection: Look under Init Containers in the describe output. Check if State: Running (process is hanging) or State: Waiting with CrashLoopBackOff.
  • Termination reason & exit code: Exit code 1 indicates runtime script failure; code 137 indicates an OOMKilled event or SIGKILL; code 127 indicates command not found.
  • Pod events section: Check if image pull errors (ErrImagePull, ImagePullBackOff) or volume mount failures are blocking the init container from even launching.
Advertisement
3️⃣

Step 2: Fetch stdout and stderr logs for the specific init container

Standard kubectl logs queries default to the first app container, which does not exist yet. You must explicitly target the init container by name.

# Target init container logs explicitly
kubectl logs payment-api-7b89df5c4-x8k9l -c wait-for-postgres

# Stream logs in real-time if init container is currently running
kubectl logs -f payment-api-7b89df5c4-x8k9l -c wait-for-postgres

# If container crashed, inspect previous instance before restart
kubectl logs payment-api-7b89df5c4-x8k9l -c wait-for-postgres --previous
  • Specify container name: Always pass -c <init-container-name>. Running kubectl logs <pod> without -c fails with container not found.
  • The --previous flag: When an init container crashes and enters backoff, inspect the logs of the crashed instance using --previous to see uncaught stack traces.
  • Common log patterns: Look for connection refused errors, database authentication timeouts, missing environment variables, or database migration syntax errors.
4️⃣

Step 3: Test network reachability, DNS, and service dependencies

A frequent root cause of stuck init containers is waiting indefinitely for an upstream service or database migration lock.

# Launch an interactive debug container in the same network namespace
kubectl debug -it payment-api-7b89df5c4-x8k9l --image=curlimages/curl --target=wait-for-postgres

# Test DNS resolution for upstream database inside cluster
nslookup postgres-service.database.svc.cluster.local

# Test TCP connectivity on database port
nc -zv -w 5 postgres-service 5432
  • CoreDNS resolution failure: Verify the init container can resolve external and in-cluster hostnames. Check if /etc/resolv.conf contains valid nameservers.
  • NetworkPolicy blockage: Check if Kubernetes NetworkPolicies or cloud security groups prevent egress from the pod's namespace to the target service.
  • Deadlocked migration locks: Database migration tools (e.g., Flyway, Liquibase, Alembic) maintain lock tables. If a previous deployment crashed mid-migration, the lock may remain held.
5️⃣

Production remediation: Implement timeouts, bounded loops, and probes

Prevent indefinite pod hangs by building defensive retry logic and failure limits into init container commands.

# Resilient init container with bounded timeout
initContainers:
- name: wait-for-db
  image: busybox:1.36
  command: ['sh', '-c']
  args:
    - |
      timeout=60
      until nc -z -w 2 postgres-service 5432 || [ $timeout -le 0 ]; do
        echo "Waiting for PostgreSQL on port 5432... ($timeout s remaining)"
        sleep 2
        timeout=$((timeout - 2))
      done
      if [ $timeout -le 0 ]; then
        echo "ERROR: Timeout reached waiting for PostgreSQL!"
        exit 1
      fi
  • Bounded timeout wrappers: Never run unbounded while true; do sleep 1; done loops. Always enforce a hard timeout that exits non-zero so Kubernetes can alert on the failure.
  • Active readiness probes on upstream: Verify dependent services have valid endpoints in kubectl get endpoints <service> before pod scheduling.
  • Resource limits on init containers: Explicitly set CPU and memory limits on init containers to prevent cluster eviction or silent node OOM panics.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: When debugging a stuck init container, use kubectl logs <pod> -c <init-container> --previous and inspect the pod's termination exit code in describe output."
⚡ 60-Second Elevator Pitch Talking Points
  • Run kubectl describe pod to inspect the init container's state, termination reason, and cluster events.
  • Fetch stdout and stderr logs by explicitly specifying the init container name with kubectl logs -c <init> --previous.
  • Verify network reachability and database migration locks, and ensure all init scripts enforce bounded timeouts.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Running kubectl logs without the -c container flag when a pod is initializing Default logs commands target the primary application container, which has not started yet, leading to confusion and empty output.
  2. Writing unbounded while-true sleep loops in init scripts without a timeout ceiling Infinite retry loops cause pods to hang silently for hours without failing, preventing automated alerting and deployment rollbacks.
  3. Overlooking OOMKilled exit codes (137) during heavy init container compilation or migrations Heavy data migrations or asset pre-compilation inside init containers can exceed memory limits and crash without logging application errors.
  4. Failing to check NetworkPolicy egress rules between application and database namespaces If egress traffic is blocked at the network policy layer, TCP health check loops in init containers will hang indefinitely.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →