Q: Your company's SLO for an API is 99.95% availability. The CTO asks, "Why not just set it to 99.99%?" What is the SRE argument against over-committing to higher SLOs?
Higher SLOs are exponentially more expensive to achieve, and over-committing creates perverse incentives.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During an on-call shift, our alerts triggered when a critical Linux production server exhibited this behavior. The interviewer is testing: SLO design philosophy, cost-reliability tradeoff, diminishing returns.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Higher SLOs are exponentially more expensive to achieve, and over-committing creates perverse incentives.
- Redundant everything: Multi-region active-active, automated failover, zero-downtime deployments
- Zero human-in-the-loop: 4.3 minutes is too short for humans to detect, diagnose, and fix—everything must be automated
- Dependency chain ceiling: If your database (e.g., RDS) offers 99.95% SLA, your service physically cannot achieve 99.99% unless you build multi-database redundancy
- Development velocity plummets (every change is a risk to the SLO)
Remediation & Permanent Safeguards
Increasing API availability from 99.9% to 99.99% reduces the allowable monthly downtime from ~43.8 minutes to just ~4.3 minutes—a 10x reduction in error budget. This leap requires fundamental architectural shifts from single-region resilience to multi-region active-active clusters with automated sub-minute failover.
- Exponential Cost Curve: Moving from three nines to four nines typically triples infrastructure costs due to multi-region data replication and networking egress.
- SRE Principle: Align SLOs with actual user pain thresholds; pursuing four nines when users are satisfied with three nines starves feature velocity.
- Redundant everything: Multi-region active-active, automated failover, zero-downtime deployments
- Zero human-in-the-loop: 4.3 minutes is too short for humans to detect, diagnose, and fix—everythi...
- Dependency chain ceiling: If your database (e.g., RDS) offers 99.95% SLA, your service physically...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when configuring companys 9995% availability SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
-
Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when configuring companys 9995% availability
Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (
df -h) shows ample free space. - Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections when configuring companys 9995% availability Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
-
Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping when configuring companys 9995% availability
Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (
pid_max).