⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 163 of 207 in Kubernetes
Senior SRE / Cloud Engineer Kubernetes Autoscaling & Resource Management Production Fire Drill
🎯 Target Role / Context: Senior SRE Incident Triage Interview

Q: HPA refuses to scale even though Prometheus shows CPU > 80%. Diagnose with cloud + K8s metrics.

Systematic triage guide for diagnosing why a Kubernetes Horizontal Pod Autoscaler fails to trigger replica scale-outs despite external Prometheus dashboards alerting on high CPU utilization.

#Kubernetes #HPA #Prometheus #Metrics Server #Autoscaling #SRE #Cloud
🎙️ Candidate Opening & Architectural Context
"During a flash sale, monitoring alerts page the on-call engineer: application CPU usage is sustained at 85% across all pods, customer latency is degrading, but the Deployment remains pinned at its minimum replica count of 3. The engineer reports that Prometheus shows the cluster on fire, but HPA refuses to scale out."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: During a flash sale, monitoring alerts page the on-call engineer: application CPU usage is sustained at 85% across all pods, customer latency is degrading, but the Deployment remains pinned at its minimum replica count of 3. The engineer reports that Prometheus shows the cluster on fire, but HPA refuses to scale out.

1️⃣

Understanding the Fundamental Decoupling: Prometheus vs Metrics Server

Kubernetes HPA does NOT read Prometheus by default. It queries the Kubernetes Metrics API (metrics.k8s.io):

  • Prometheus collects metrics asynchronously via scraping agents (Node Exporter, cAdvisor).
  • HPA relies on metrics-server scraping kubelet summary APIs every 15–60 seconds.
  • If metrics-server is degraded, failing TLS verification, or hitting API limits, HPA is blinded.
2️⃣

Execute Diagnostic CLI Triage

Inspect HPA conditions, resource definitions, and metrics API health:

# 1. Check HPA status and conditions
kubectl describe hpa <hpa-name>

# Look for:
# Conditions:
#   AbleToScale: True
#   ScalingActive: False (FailedGetResourceMetric)
# Current: <unknown> / 80%

# 2. Check if metrics-server is serving metrics
kubectl top pods -l app=<app-name>
kubectl get apiservice v1beta1.metrics.k8s.io

# 3. Check container resources specification
kubectl get deployment <app-name> -o yaml | grep -A 8 resources
  • Missing CPU Requests: If resources.requests.cpu is omitted, HPA calculation is mathematically impossible. HPA computes target percentage as: (actual usage / requested CPU) * 100. Without requests, HPA shows <unknown>.
  • Max Replicas Reached: Check if current replicas == spec.maxReplicas.
  • Stabilization Window: Check if downscale/upscale stabilization windows are throttling scaling actions.
Advertisement
3️⃣

Check Node Capacity & Cluster Autoscaler Blockades

If HPA updated desiredReplicas but pods cannot be scheduled:

kubectl get pods -l app=<app-name> | grep Pending
kubectl describe pod <pending-pod> | grep -A 5 Events
# Look for: "0/12 nodes are available: 12 Insufficient cpu."
  • If nodes are fully packed and the Cloud Cluster Autoscaler / Karpenter is blocked by AWS EC2 quota limits or subnet IP exhaustion, pods remain stuck in Pending.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"HPA scaling failures almost always boil down to three root causes: missing container CPU requests, metrics-server failure, or cluster capacity constraints blocking Pending pods."
⚡ 60-Second Elevator Pitch Talking Points
  • HPA does not read Prometheus by default; it reads the Kubernetes Metrics API (metrics-server). First, I run 'kubectl describe hpa' to check the ScalingActive condition.
  • If HPA shows '<unknown> / 80%', the most common cause is that the container specification lacks 'resources.requests.cpu', making percentage calculation mathematically undefined.
  • Second, I check if metrics-server is healthy using 'kubectl get apiservice v1beta1.metrics.k8s.io' and 'kubectl top pods'.
  • Third, I verify whether HPA has already hit 'maxReplicas', or if upscale stabilization windows are suppressing new events.
  • Finally, if desired replicas increased but actual pods are stuck in Pending, I inspect Cluster Autoscaler / Karpenter logs for EC2 quota or subnet IP exhaustion.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Over-granting cluster-admin permissions to pod ServiceAccounts instead of scoping namespace Roles Over-granting cluster-wide permissions violates least-privilege principles and allows compromised pods to escalate privileges across namespaces.
  2. Referencing mutable latest container tags across production Kubernetes Deployment specs Deployments referencing :latest risk inconsistent versions across replica pods and break deterministic rollback capabilities.
  3. Omitting pod readiness probes, causing traffic to hit uninitialized application containers Readiness probes gate traffic delivery; missing probes cause Kubernetes to route user requests to pods before dependencies or caches are loaded.
  4. Executing diagnostic triage without analyzing chronological cluster events and describe outputs Cluster events provide sequential timestamps of scheduler decisions, probe failures, and image pull timeouts essential for rapid root-cause isolation.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →