Q: A database server shows 100% disk utilization on iostat, but random read latency is acceptable while sequential write latency spikes wildly. How do you distinguish between read bottlenecks and write bottlenecks, and what tools reveal the root cause?
Disk latency has two distinct bottlenecks: read latency (affecting query performance) and write latency (affecting commits/fsync). They require different tools to diagnose.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain my systematic Linux troubleshooting methodology using Brendan Gregg's USE method. The interviewer is testing: Advanced I/O profiling, disk subsystem diagnostics, queue depth.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Disk latency has two distinct bottlenecks: read latency (affecting query performance) and write latency (affecting commits/fsync). They require different tools to diagnose.
r/sandw/s: Read and write operations per second (independently measured).rareq-szandwareq-sz: Average request sizes (small vs large).r_awaitandw_await: Average latency for reads and writes (in ms).svctm: Service time (how long the disk takes to serve one request).%util: Percentage of time the disk is busy (100% = fully saturated).
Remediation & Permanent Safeguards
Tool 1: iostat -x 1 shows: If r_await is 2ms but w_await is 500ms, writes are the bottleneck, not reads. Tool 2: blktrace + blkparse captures every disk I/O operation: Output shows timestamp, operation type (R/W), block address, size, queue depth, and latency for each I/O. You can see if writes are getting stuck behind a large read or being reordered by the scheduler. Root causes:
blktrace -d /dev/sda -o - | blkparse -i -
- High write latency + queue depth > 1: Disk controller or SSD firmware throttling writes (e.g., NAND garbage collection).
- High write latency + queue depth = 1: Single slow write (e.g., journal flush to slow media or encryption overhead).
- Random writes slower than sequential: RAID write-back cache eviction or lack of RAID optimization for random workloads.
- r/s and w/s: Read and write operations per second (independently measured).
- rareq-sz and wareq-sz: Average request sizes (small vs large).
- r_await and w_await: Average latency for reads and writes (in ms).
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting an asynchronous database read replica without confirming replication lag and WAL synchronization Promoting an asynchronous replica that lags behind the primary causes transaction loss and data inconsistency.
-
Connecting distributed application microservices directly without a proxy connection pooler (PgBouncer)
Direct connections create thread-per-client overhead, consuming database memory and hitting
max_connectionslimits under sudden traffic spikes. - Overlooking WAL and binary log volume growth on database instances during intensive batch write jobs Unchecked replication or unpurged transaction logs rapidly exhaust disk space, placing the database into read-only recovery mode.
- Neglecting row-level lock contention and lock ordering across concurrent distributed transactions in linux-sre-q046-a-database-server-shows-100-percent-disk-u Inconsistent lock ordering across distributed workers causes cyclic lock waits, resulting in aborted transactions and high error rates.