⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 8 of 135 in Linux & SRE
Senior DevOps / SRE [L2] General DevOps General DevOps — Scenario-Based Interview Questions Production Scenario [L2]
🎯 Target Role / Context: General DevOps Scenario #2 (L2) · General DevOps — Scenario-Based Interview Questions

Q: Your team manages an application that writes millions of logs to disk each hour. Suddenly, all disks across the fleet fill up simultaneously, causing a massive outage. What is the fundamental architecture flaw, and how do you fix it?

The architectural flaw is storing unbounded state (logs) on the application instance's local filesystem without rotation.

#General DevOps #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: Log rotation, log shipping, decoupling state.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: Log rotation, log shipping, decoupling state.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Initial Diagnostics & Root Cause Analysis

The architectural flaw is storing unbounded state (logs) on the application instance's local filesystem without rotation.

  • Log Rotation: Immediately configure logrotate to compress and delete logs older than a few hours or when they exceed a certain MB threshold. This caps the maximum disk usage.
  • Decoupling State: In modern DevOps (e.g., Docker/Kubernetes), applications shouldn't manage log files on disk at all. Applications should log purely to STDOUT and STDERR. A DaemonSet or sidecar agent (like Fluentbit or Filebeat) acts as a pipe, reading those streams and shipping them entirely off the host to a centralized aggregator (Datadog/Elasticsearch) preventing local disk exhaustion completely.
2️⃣

Remediation & Permanent Safeguards

An SRE fix involves:

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Log Rotation: Immediately configure logrotate to compress and delete logs older than a few hours or when they exceed a certain MB ."
⚡ 60-Second Elevator Pitch Talking Points
  • Log Rotation: Immediately configure logrotate to compress and delete logs older than a few hours ...
  • Decoupling State: In modern DevOps (e.g., Docker/Kubernetes), applications shouldn't manage log f...
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring General team manages Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring General team manages Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring General team manages Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring General team manages Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →