⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Databases & Storage Interview Questions Scenario 19 of 54 in Databases & Storage
Staff SRE / Principal Architect [L3] Database Linux / SRE — Scenario-Based Interview Questions Staff SRE Scenario [L3]
🎯 Target Role / Context: Linux Scenario #46 (L3) · Linux / SRE — Scenario-Based Interview Questions

Q: A database server shows 100% disk utilization on iostat, but random read latency is acceptable while sequential write latency spikes wildly. How do you distinguish between read bottlenecks and write bottlenecks, and what tools reveal the root cause?

Disk latency has two distinct bottlenecks: read latency (affecting query performance) and write latency (affecting commits/fsync). They require different tools to diagnose.

#Linux #L3 #SRE #Systems #Troubleshooting
🎙️ Candidate Opening & Architectural Context
""In an interview, I explain my systematic Linux troubleshooting methodology using Brendan Gregg's USE method. The interviewer is testing: Advanced I/O profiling, disk subsystem diagnostics, queue depth.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "In an interview, I explain my systematic Linux troubleshooting methodology using Brendan Gregg's USE method. The interviewer is testing: Advanced I/O profiling, disk subsystem diagnostics, queue depth.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Initial Diagnostics & Root Cause Analysis

Disk latency has two distinct bottlenecks: read latency (affecting query performance) and write latency (affecting commits/fsync). They require different tools to diagnose.

  • r/s and w/s: Read and write operations per second (independently measured).
  • rareq-sz and wareq-sz: Average request sizes (small vs large).
  • r_await and w_await: Average latency for reads and writes (in ms).
  • svctm: Service time (how long the disk takes to serve one request).
  • %util: Percentage of time the disk is busy (100% = fully saturated).
2️⃣

Remediation & Permanent Safeguards

Tool 1: iostat -x 1 shows: If r_await is 2ms but w_await is 500ms, writes are the bottleneck, not reads. Tool 2: blktrace + blkparse captures every disk I/O operation: Output shows timestamp, operation type (R/W), block address, size, queue depth, and latency for each I/O. You can see if writes are getting stuck behind a large read or being reordered by the scheduler. Root causes:

blktrace -d /dev/sda -o - | blkparse -i -
  • High write latency + queue depth > 1: Disk controller or SSD firmware throttling writes (e.g., NAND garbage collection).
  • High write latency + queue depth = 1: Single slow write (e.g., journal flush to slow media or encryption overhead).
  • Random writes slower than sequential: RAID write-back cache eviction or lack of RAID optimization for random workloads.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: r/s and w/s: Read and write operations per second (independently measured).."
⚡ 60-Second Elevator Pitch Talking Points
  • r/s and w/s: Read and write operations per second (independently measured).
  • rareq-sz and wareq-sz: Average request sizes (small vs large).
  • r_await and w_await: Average latency for reads and writes (in ms).
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting an asynchronous database read replica without confirming replication lag and WAL synchronization Promoting an asynchronous replica that lags behind the primary causes transaction loss and data inconsistency.
  2. Connecting distributed application microservices directly without a proxy connection pooler (PgBouncer) Direct connections create thread-per-client overhead, consuming database memory and hitting max_connections limits under sudden traffic spikes.
  3. Overlooking WAL and binary log volume growth on database instances during intensive batch write jobs Unchecked replication or unpurged transaction logs rapidly exhaust disk space, placing the database into read-only recovery mode.
  4. Neglecting row-level lock contention and lock ordering across concurrent distributed transactions in linux-sre-q046-a-database-server-shows-100-percent-disk-u Inconsistent lock ordering across distributed workers causes cyclic lock waits, resulting in aborted transactions and high error rates.
Advertisement
Want more Databases & Storage scenarios?
Explore our complete collection of scenario-based Databases & Storage interview runbooks.
Browse All Databases & Storage Questions →