Q: A cluster-autoscaler is not scaling up even though pods are Pending. What could be wrong?
Pod is unschedulable for a reason other than resources — e.g., node affinity requires a specific label that no node type has. CA won't add nodes it can't schedule the pod on.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
Check: CA logs — kubectl logs -n kube-system — it logs exactly why it's not scaling.
- Pod is unschedulable for a reason other than resources — e.g., node affinity requires a specific label that no node type has. CA won't add nodes it can't schedule the pod on.
- Max node count reached — CA has a configured max.
--max-nodes-totalor per-node-group limit. - Pod has
cluster-autoscaler.kubernetes.io/safe-to-evict: false— CA may refuse to scale if eviction of existing pods is blocked. - Cooldown period — CA has a scale-up cooldown (default 10 min). May be waiting.
Remediation & Permanent Safeguards
- Budget exhausted — cloud account has hit EC2/VM quota.
- CA can't provision the requested instance type — spot capacity unavailable.
- Pod is unschedulable for a reason other than resources — e.g., node affinity requires a specific ...
- Max node count reached — CA has a configured max. --max-nodes-total or per-node-group limit.
- Pod has cluster-autoscaler.kubernetes.io/safe-to-evict: false — CA may refuse to scale if evictio...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when configuring cluster autoscaler scaling Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes when configuring cluster autoscaler scaling
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending when configuring cluster autoscaler scaling
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods when configuring cluster autoscaler scaling Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.