Q: A pod fails to start with 'CreateContainerError'. There are no container logs available via 'kubectl logs'. What causes this error before application code can execute, and how do you differentiate container runtime sandbox creation errors from volume mount deadlocks and security policy denials?

Inspect the pod status and event timeline to capture the exact failure string returned by the container runtime:

• Describe the Pod: Run kubectl describe pod <pod-name> and scroll to the Containers -> State block. Check Reason: CreateContainerError and read the Message field.
• Extract Events: Run kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by=.metadata.creationTimestamp.
• Classify Failure Category: The error message will pinpoint whether the failure is in MountVolume.SetUp failed, failed to generate spec, permission denied, or OCI runtime create failed.

The most frequent cause of CreateContainerError in cloud environments is a storage volume that cannot be mounted into the container namespace:

• Multi-Attach Error: Check if a PersistentVolume (e.g. AWS EBS ReadWriteOnce) is still locked to another worker node: kubectl get volumeattachments.
• Invalid subPath in VolumeMount: If using volumeMounts.subPath, verify that the subPath actually exists in the persistent volume or that Kubelet has permission to create it.
• Missing ConfigMap/Secret File Mount: If a volume mounts a ConfigMap or Secret that was deleted after pod scheduling, container creation fails during the bind mount.

Linux security modules will block container runtime initialization if an annotated profile is missing from the node:

• Audit AppArmor Annotations: Look for container.apparmor.security.beta.kubernetes.io/<container>: localhost/custom-profile. If custom-profile is not loaded in the node's /etc/apparmor.d/, containerd aborts container creation.
• Check Node Kernel Audit Logs: On the worker node, run dmesg -T | grep -i 'apparmor\|selinux\|audit' to locate rejected capability bindings.
• Verify Seccomp Profile Path: Verify if securityContext.seccompProfile.localhostProfile points to a valid file inside /var/lib/kubelet/seccomp/.

If the worker node has exhausted OS resources, containerd cannot allocate new kernel namespace descriptors:

• Check Inotify Watchers: Microservices with file watchers can exhaust fs.inotify.max_user_watches. Check usage via: cat /proc/sys/fs/inotify/max_user_watches.
• Check Node Kernel Task Limits: Run cat /proc/sys/kernel/pid_max and compare with current running threads: ps -eLf | wc -l.
• Inspect Containerd Service Logs: Run journalctl -u containerd -n 50 --no-pager for too many open files or failed to allocate cgroup.

Execute targeted remediations based on the identified failure vector:

• Force Release Stuck EBS Volume: If volume attachment is stuck, delete the blocking volumeattachment object or detach the volume via AWS CLI: aws ec2 detach-volume --volume-id <vol-id> --force.
• Fix subPath Missing Directory: Add an initContainer to create the necessary directory structure on the PV before the main container attempts to mount it.
• Adjust Host Inotify Limits: Permanently increase inotify watches in /etc/sysctl.d/99-kubernetes.conf: fs.inotify.max_user_watches = 524288.

Verify the container reaches Running state and practice hands-on debugging in the lab:

• Verify Pod Status: Confirm STATUS: Running with kubectl get pod <pod-name>.
• Check Container Runtime Health: Run crictl ps --name <pod-name> on the node.
• Lab Runbook: Practice reproducing and fixing CreateContainerError in DevOps Lab: CreateContainerError Hands-On Runbook ↗.