Q: A production database server uses a hardware RAID-10 array. One disk fails, and the RAID begins rebuilding onto a hot spare. During the rebuild, I/O performance degrades severely. Why does this happen, and how do you mitigate it?

During a RAID rebuild, the controller must read data from the surviving disks to reconstruct the missing disk's data onto the hot spare. This rebuild I/O competes directly with production I/O for the same physical disk heads, bus bandwidth, and controller queue slots.

• Read amplification: Every block on the surviving disks must be read to reconstruct parity/mirror data—even blocks the application doesn't need.
• Contention: The rebuild process and production queries fight for disk seek time, especially on spinning HDDs.
• Write penalty: RAID-10 mirrors writes, so the rebuild adds extra write operations.
• Throttle rebuild speed: Most RAID controllers allow setting rebuild priority (e.g., megacli -AdpSetProp RebuildRate -val 30 -a0 sets it to 30%). Lower values reduce rebuild impact but increase the vulnerable window.

Why performance degrades: Mitigation strategies:

• Schedule rebuilds during low traffic: If possible, initiate manual rebuilds during off-peak hours.
• Use SSDs: SSDs eliminate seek-time contention, making rebuild impact nearly negligible.
• RAID controller cache: Ensure the battery-backed write cache is functioning—it absorbs write bursts during rebuild.