Q: An application is occasionally utilizing 100% CPU, but the spike only lasts for 3 seconds every hour, making it impossible to confidently run `perf` or `top` in time to catch it live. How do you identify the exact function causing the spike?
I would implement Continuous Profiling.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain how we designed actionable, symptom-based alerting using the Four Golden Signals. The interviewer is testing: Continuous Profiling in Production (e.g., Pyroscope, Datadog Profiler).. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
I would implement Continuous Profiling. Traditional profiling introduces massive overhead and is run manually ad-hoc. Continuous profilers (like Pyroscope or Datadog Continuous Profiler) run permanently in production utilizing extremely low-overhead eBPF or sampling techniques (e.g., capturing the stack trace only 100 times a second). When the 3-second spike happens, the profiler automatically records it. The next morning, I can review the profiler's UI, select the exact 5-minute slice surrounding the spike, and look at the generated Flamegraph, which visualizes exactly which functions or lines of code consumed the CPU cycles across the entire fleet retroactively.
- Immediate Triage: I would implement Continuous Profiling.
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability application occasionally High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability application occasionally Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability application occasionally Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability application occasionally Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.