Q: After a Kubernetes upgrade, Prometheus shows `up == 0` for many pod scrape targets. What do you check?
I would troubleshoot the scrape path from Prometheus to the pods.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During a high-traffic production event, our observability stack proved essential in isolating this latency surge. The interviewer is testing: Kubernetes service discovery, scraping, network and auth issues.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
I would troubleshoot the scrape path from Prometheus to the pods.
- Service discovery: Are the pods still discovered with the expected labels and annotations?
- Endpoint changes: Did ServiceMonitor, PodMonitor, or scrape configs stop matching after label changes?
- NetworkPolicy: Can Prometheus still reach pod IPs and metrics ports?
- TLS/auth: Did certificates, service account tokens, or mTLS settings change?
Remediation & Permanent Safeguards
up == 0 is a scrape failure. The application may be healthy, but Prometheus cannot collect its metrics.
- Metrics endpoint: Does
/metricsstill respond from inside the cluster? - Prometheus logs: Look for scrape errors such as timeout, connection refused, 401, 403, or x509 failures.
- Service discovery: Are the pods still discovered with the expected labels and annotations?
- Endpoint changes: Did ServiceMonitor, PodMonitor, or scrape configs stop matching after label cha...
- NetworkPolicy: Can Prometheus still reach pod IPs and metrics ports?
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability Kubernetes upgrade High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability Kubernetes upgrade Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability Kubernetes upgrade Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability Kubernetes upgrade Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.