Q: Your log aggregator is consuming massive amounts of AWS storage costs because it retains all logs for 30 days. You need to keep 30 days of data for forensics, but cut costs deeply. What is the standard architectural design?
SRE teams must implement a multi-tier storage architecture, often called Hot/Warm/Cold tiers.
#Observability #Observability #L3 #Monitoring #Prometheus #SRE
🎙️ Candidate Opening & Architectural Context
""In an interview, I explain how we designed actionable, symptom-based alerting using the Four Golden Signals. The interviewer is testing: Log lifecycle management, cold storage architectures.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
SRE teams must implement a multi-tier storage architecture, often called Hot/Warm/Cold tiers.
- Hot Tier: Keep only 3-7 days of logs in the expensive, fast-SSD log aggregator (e.g., Elasticsearch or Datadog) for immediate incident response and daily dashboarding.
- Cold Tier / Archive: Use a routing layer (like fluentbit or logstash) to tee all raw log data concurrently to a cheap AWS S3 bucket as gzipped JSON files.
2️⃣
Remediation & Permanent Safeguards
If a forensic audit is required 25 days later, the SRE queries the S3 bucket directly using AWS Athena (Presto over S3) without needing to pay the premium to keep that data instantly indexed in the hot tier.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Hot Tier: Keep only 3-7 days of logs in the expensive, fast-SSD log aggregator (e.g., Elasticsearch or Datadog) for immediate inci."
⚡ 60-Second Elevator Pitch Talking Points
- Hot Tier: Keep only 3-7 days of logs in the expensive, fast-SSD log aggregator (e.g., Elasticsear...
- Cold Tier / Archive: Use a routing layer (like fluentbit or logstash) to tee all raw log data con...
Advertisement