Q: What is the difference between a Push-based monitoring system (like DataDog/StatsD) and a Pull-based system (like Prometheus)?
Push: The application (or an agent on the host) writes metrics actively and sends them over the network to a centralized aggregator endpoint (e.g., Datadog, InfluxDB).
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Logs tell you what happened, metrics tell you where to look, and distributed traces pinpoint the exact slow component. The interviewer is testing: Architecture, network topologies, auto-discovery.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
*Pros:* Easier to span NATs or firewalls (outbound is usually allowed), great for ephemeral/serverless functions that die too fast to be scraped.
- Push: The application (or an agent on the host) writes metrics actively and sends them over the network to a centralized aggregator endpoint (e.g., Datadog, InfluxDB).
- Pull (Prometheus): The centralized server uses an HTTP GET request (scrape) to pull a
/metricsendpoint exposed by the application.
Remediation & Permanent Safeguards
*Cons:* Can overwhelm the central server with UDP floods, and the aggregator doesn't inherently know if an agent died vs simply has no data to send. *Pros:* The server controls the ingestion rate, preventing DDOS. It implicitly knows when a service is dead because the HTTP GET fails (up == 0). It heavily relies on Service Discovery (like Consul or Kubernetes API) to find targets dynamically.
- Push: The application (or an agent on the host) writes metrics actively and sends them over the n...
- Pull (Prometheus): The centralized server uses an HTTP GET request (scrape) to pull a /metrics en...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability difference Push High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability difference Push Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability difference Push Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability difference Push Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.