Q: During a high-traffic production event, an alert triggers: multiple worker nodes enter 'NotReady' status. Pods start evicting or hanging in Terminating state, and cluster capacity drops. Walk me through your step-by-step triage workflow to identify the root cause (kubelet crash, containerd freeze, TLS cert expiration, or CNI failure) and bring the node back to Ready without dropping in-flight traffic.

Inspect node conditions and active Lease objects to determine how long the node has been unresponsive and why the Node Lifecycle Controller flagged it:

• Extract Node Conditions: Run kubectl describe node <node-name> and inspect the Conditions table. Check if Ready=False or Ready=Unknown, and note the Reason (e.g. KubeletNotReady, ContainerRuntimeUnresponsive).
• Inspect Node Heartbeat Lease: Run kubectl get leases -n kube-node-lease <node-name> -o yaml to verify the renewTime timestamp. If the renewTime is older than node-monitor-grace-period (40s default), kubelet has lost contact with the control plane.
• Check Pod Impact: Verify if pods on the node are transitioning to Terminating or NodeLost: kubectl get pods -A --field-selector spec.nodeName=<node-name> -o wide.

SSH into the troubled host node (or launch an AWS SSM session) and investigate the Kubelet service:

• Check Kubelet Process Status: Run systemctl status kubelet. Look for failed, activating (auto-restart), or crashed state.
• Inspect Kubelet Systemd Logs: Run journalctl -u kubelet -n 120 --no-pager to locate recent fatal log messages.
• Audit Cgroup Driver Configuration: Verify if Docker or systemd recently updated and caused a cgroup mismatch. Both Kubelet (/var/lib/kubelet/config.yaml) and Containerd (/etc/containerd/config.toml) must agree on SystemdCgroup = true.
• Check Disk / Inode & PID Capacity: Verify system-level OS limits: df -h, df -i, and sysctl kernel.pid_max. If PID exhaustion occurred, systemd cannot fork kubelet health checks.

Kubelet cannot stay Ready if the underlying CRI container runtime is frozen, socket-locked, or deadlocked:

• Inspect Containerd Service: Run systemctl status containerd. Verify the process is active and not hung in D-state (uninterruptible disk sleep).
• Probe the CRI Socket via crictl: Run crictl info and crictl ps. If crictl hangs or throws rpc error: code = Unavailable desc = connection error: desc = 'transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused', containerd is deadlocked.
• Inspect Containerd Logs: Check journalctl -u containerd -n 100 --no-pager for storage snapshotter panics, overlayfs corruption, or high I/O wait.
• Restart Sequence: If deadlocked: systemctl restart containerd && systemctl restart kubelet.

When kubelet client certificates expire, the kubelet cannot authenticate to the API server, causing the node to flip to NotReady without any application crashes:

• Check Active Certificate Validity: Run openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates.
• Check Auto-Rotation Configuration: Inspect /var/lib/kubelet/config.yaml to verify rotateCertificates: true is present under the KubeletConfiguration.
• Inspect Pending CSRs: On the control plane / bastion, run kubectl get csr. If a certificate signing request for the node is stuck in Pending, manually approve it via kubectl certificate approve <csr-name>.

If kubelet cannot establish network interfaces for pods or communicate with the control plane, the Ready condition will be set to False with NetworkPluginNotReady:

• Check CNI DaemonSet Pods: Run kubectl get pods -n kube-system -o wide | grep <node-name> for AWS VPC CNI (aws-node), Calico, or Cilium.
• Check CNI Log Files: On the host, inspect /var/log/aws-routed-eni/ipamd.log or /var/log/cilium/cilium.log. For AWS EKS, verify if the node ran out of available secondary private IPs on its ENIs.
• Test API Server Egress: From the node, run curl -k https://<kubernetes-api-endpoint>:443/version to confirm security groups and route tables allow bidirectional traffic.

Once the root cause is resolved and the node condition flips back to Ready, safely reintegrate it into cluster traffic:

• Verify Node Ready Status: Run kubectl get node <node-name> and confirm STATUS: Ready.
• Uncordon the Node: Execute kubectl uncordon <node-name> to allow new workloads to schedule.
• Verify Kubelet Telemetry: Check Prometheus / Datadog node metrics for CPU, RAM, disk I/O, and kubelet API request latency.
• Hands-On Lab Runbook: Practice step-by-step Node Not Ready recovery in our hands-on lab at DevOps Lab: Node Not Ready Hands-On Runbook ↗.