Q: A pod transitions into 'RunContainerError'. Explain the exact difference between RunContainerError and CrashLoopBackOff. What operating system kernel and runtime events trigger RunContainerError, and how do you triage executable permissions, missing shells in Alpine/Distroless containers, and failing postStart hooks?

Understand the precise kernel boundary that differentiates these two failure modes:

• RunContainerError: The host kernel execve(2) system call failed to execute the command/entrypoint, or a postStart lifecycle hook exited with a non-zero status. The container never entered 'Running' state.
• CrashLoopBackOff: The binary was found and executed by the kernel. The application initialized (PID 1 ran), but subsequently terminated with an exit code (e.g. exit 1 due to database connection failure).
• Check Pod State: In kubectl get pod, the container state will display Waiting with Reason: RunContainerError.

The most frequent trigger is an invalid entrypoint path, missing interpreter, or Windows line endings:

• Examine Event Details: kubectl describe pod <pod-name> will display:
  failed to create containerd task: failed to start shim: exec: "/entrypoint.sh": stat /entrypoint.sh: no such file or directory.
• The Windows CRLF Trap: If an entrypoint script was edited on Windows, it contains carriage returns (\r\n). When Linux reads #!/bin/sh\r, it attempts to find the interpreter /bin/sh\r and fails with 'no such file or directory' even though /bin/sh exists!
• Fix with dos2unix: Convert line endings before building: dos2unix entrypoint.sh.

Modern minimal security base images do not include /bin/bash or /bin/sh:

• The Distroless Trap: In Google Distroless or scratch images, there is no shell. If a pod manifest specifies command: ["/bin/bash", "-c", "..."], containerd fails with exec: "/bin/bash": stat /bin/bash: no such file or directory.
• The Alpine musl vs glibc Trap: Alpine uses musl libc. A Go or C++ binary dynamically compiled against glibc on Ubuntu will fail to execute in Alpine with no such file or directory because the dynamic linker (/lib64/ld-linux-x86-64.so.2) is absent.
• Fix: Build static binaries (CGO_ENABLED=0) or execute the binary directly without wrapping in a shell: command: ["/app/server"].

Linux permissions and Kubernetes SecurityContext restrictions preventing execution:

• Missing Execution Bit: If entrypoint.sh lacks execute permissions, containerd returns permission denied. Ensure Dockerfile includes: RUN chmod +x /entrypoint.sh.
• runAsNonRoot Violation: When securityContext.runAsNonRoot: true is enforced, and the container image sets USER 0 (root) or omits the USER directive, Kubelet refuses to run the container and flags RunContainerError.
• Privileged Port Binding: If a non-root user (e.g. UID 1000) tries to bind port 80/443 without CAP_NET_BIND_SERVICE capability, the kernel aborts.

Kubernetes lifecycle hooks run asynchronously alongside the container entrypoint, but with a strict guarantee:

• The postStart Contract: If a postStart hook handler (either exec command or HTTP GET) fails with a non-zero exit code, Kubernetes kills the container immediately and marks the pod with RunContainerError.
• Audit Pod Spec: Inspect lifecycle.postStart in the deployment manifest.
• Inspect Hook Failure in Events: Run kubectl describe pod | grep -i poststart. Look for FailedPostStartHook: Exec lifecycle hook ([/bin/sh -c ...]) for Container failed.
• Fix: Ensure postStart scripts do not fail silently, or migrate initialization tasks into dedicated initContainers.

Test the fix using an ephemeral debug container or local docker run:

• Local Verification: Run locally: docker run --rm -it <image> /bin/sh to test file permissions and shebangs.
• Ephemeral Debugging in K8s: Attach a debug container: kubectl debug -it <pod-name> --image=busybox --target=<container>.
• Lab Runbook: Practice fixing RunContainerError, distroless shell traps, and postStart hooks in DevOps Lab: RunContainerError Hands-On Runbook ↗.