Q: Your init container is stuck and the main container never starts. How do you debug?
Debug a stuck Kubernetes init container by inspecting pod events with describe, checking init container logs, and verifying blocking service dependencies.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When an init container is stuck, the Kubernetes kubelet halts pod initialization because init containers must complete sequentially and exit with code 0 before app containers start. The pod status remains frozen in <code>Init:0/1</code>, <code>Init:CrashLoopBackOff</code>, or <code>PodInitializing</code>. To diagnose the failure quickly, senior engineers follow a structured triage path: check pod status and termination exit codes, extract init container logs, test network dependencies, and resolve deadlocks.
Triage Why Your Init Container is Stuck Using Kubectl Logs
Init containers execute sequentially in the exact order declared under spec.initContainers. Understanding status codes is critical for triage.
- Init:N/M status: In a pod with M init containers, N have successfully completed with exit code 0. If stuck at
Init:0/1, the first init container is either hanging or actively running. - Init:CrashLoopBackOff: The init container completed with a non-zero exit code. Kubelet restarts it with an exponential backoff delay (10s, 20s, up to 5m).
- Main container dormancy: The primary application containers in
spec.containersremain inWaitingstate with reasonPodInitializinguntil all init containers succeed.
Step 1: Inspect pod status, container exit codes, and recent events
Execute kubectl describe to extract the init container's current state, last termination reason, exit code, and cluster events.
# Identify pods stuck in Init state
kubectl get pods -l app=payment-api
# Detailed inspection of container state and events
kubectl describe pod payment-api-7b89df5c4-x8k9l
# Extract exact state and exit codes via jsonpath
kubectl get pod payment-api-7b89df5c4-x8k9l -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'
- State inspection: Look under
Init Containersin the describe output. Check ifState: Running(process is hanging) orState: WaitingwithCrashLoopBackOff. - Termination reason & exit code: Exit code
1indicates runtime script failure; code137indicates an OOMKilled event or SIGKILL; code127indicates command not found. - Pod events section: Check if image pull errors (
ErrImagePull,ImagePullBackOff) or volume mount failures are blocking the init container from even launching.
Step 2: Fetch stdout and stderr logs for the specific init container
Standard kubectl logs queries default to the first app container, which does not exist yet. You must explicitly target the init container by name.
# Target init container logs explicitly
kubectl logs payment-api-7b89df5c4-x8k9l -c wait-for-postgres
# Stream logs in real-time if init container is currently running
kubectl logs -f payment-api-7b89df5c4-x8k9l -c wait-for-postgres
# If container crashed, inspect previous instance before restart
kubectl logs payment-api-7b89df5c4-x8k9l -c wait-for-postgres --previous
- Specify container name: Always pass
-c <init-container-name>. Runningkubectl logs <pod>without-cfails withcontainer not found. - The --previous flag: When an init container crashes and enters backoff, inspect the logs of the crashed instance using
--previousto see uncaught stack traces. - Common log patterns: Look for connection refused errors, database authentication timeouts, missing environment variables, or database migration syntax errors.
Step 3: Test network reachability, DNS, and service dependencies
A frequent root cause of stuck init containers is waiting indefinitely for an upstream service or database migration lock.
# Launch an interactive debug container in the same network namespace
kubectl debug -it payment-api-7b89df5c4-x8k9l --image=curlimages/curl --target=wait-for-postgres
# Test DNS resolution for upstream database inside cluster
nslookup postgres-service.database.svc.cluster.local
# Test TCP connectivity on database port
nc -zv -w 5 postgres-service 5432
- CoreDNS resolution failure: Verify the init container can resolve external and in-cluster hostnames. Check if
/etc/resolv.confcontains valid nameservers. - NetworkPolicy blockage: Check if Kubernetes NetworkPolicies or cloud security groups prevent egress from the pod's namespace to the target service.
- Deadlocked migration locks: Database migration tools (e.g., Flyway, Liquibase, Alembic) maintain lock tables. If a previous deployment crashed mid-migration, the lock may remain held.
Production remediation: Implement timeouts, bounded loops, and probes
Prevent indefinite pod hangs by building defensive retry logic and failure limits into init container commands.
# Resilient init container with bounded timeout
initContainers:
- name: wait-for-db
image: busybox:1.36
command: ['sh', '-c']
args:
- |
timeout=60
until nc -z -w 2 postgres-service 5432 || [ $timeout -le 0 ]; do
echo "Waiting for PostgreSQL on port 5432... ($timeout s remaining)"
sleep 2
timeout=$((timeout - 2))
done
if [ $timeout -le 0 ]; then
echo "ERROR: Timeout reached waiting for PostgreSQL!"
exit 1
fi
- Bounded timeout wrappers: Never run unbounded
while true; do sleep 1; doneloops. Always enforce a hard timeout that exits non-zero so Kubernetes can alert on the failure. - Active readiness probes on upstream: Verify dependent services have valid endpoints in
kubectl get endpoints <service>before pod scheduling. - Resource limits on init containers: Explicitly set CPU and memory limits on init containers to prevent cluster eviction or silent node OOM panics.
- Run kubectl describe pod to inspect the init container's state, termination reason, and cluster events.
- Fetch stdout and stderr logs by explicitly specifying the init container name with kubectl logs -c <init> --previous.
- Verify network reachability and database migration locks, and ensure all init scripts enforce bounded timeouts.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Running kubectl logs without the -c container flag when a pod is initializing Default logs commands target the primary application container, which has not started yet, leading to confusion and empty output.
- Writing unbounded while-true sleep loops in init scripts without a timeout ceiling Infinite retry loops cause pods to hang silently for hours without failing, preventing automated alerting and deployment rollbacks.
- Overlooking OOMKilled exit codes (137) during heavy init container compilation or migrations Heavy data migrations or asset pre-compilation inside init containers can exceed memory limits and crash without logging application errors.
- Failing to check NetworkPolicy egress rules between application and database namespaces If egress traffic is blocked at the network policy layer, TCP health check loops in init containers will hang indefinitely.