Staff SRE / Principal Architect [L3] Linux Linux / SRE — Scenario-Based Interview Questions Staff SRE Scenario [L3]

Q: The OS memory is exhausted, but `top` and `ps` show that the sum of all running applications is only using 20% of the RAM. You clear the page cache, but the memory is still gone. Where is the "missing" memory?

Q: The OS memory is exhausted, but `top` and `ps` show that the sum of all running applications is only using 20% of the RAM. You clear the page cache, but the memory is still gone. Where is the \"missing\" memory?

\"We encountered this OS-level bottleneck during peak traffic and diagnosed it down to kernel and filesystem metrics. The interviewer is testing: Kernel Slab allocations, dentry/inode caches, memory leaks in kernel modules.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.\" Key Strategy: Pro-Tip: If user-space processes and the page cache aren't holding the memory, it is allocated directly within the Kernel Slab Allocator. T.

If user-space processes and the page cache aren't holding the memory, it is allocated directly within the Kernel Slab Allocator. The Slab...

#Linux #Linux / SRE — Scenario-Based Interview Questions #L3 #SRE #Systems #Troubleshooting

🎙️ Candidate Opening & Architectural Context

""We encountered this OS-level bottleneck during peak traffic and diagnosed it down to kernel and filesystem metrics. The interviewer is testing: Kernel Slab allocations, dentry/inode caches, memory leaks in kernel modules.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Production Solution & Architecture

If user-space processes and the page cache aren't holding the memory, it is allocated directly within the Kernel Slab Allocator. The Slab holds kernel structures like active dentry caches (directory paths) or inodes, or it's a literal memory leak caused by a buggy kernel driver. I would run the slabtop command. If slabtop shows the dentry cache is gigabytes in size, the server has traversed millions of tiny files recently. While normally the system reclaims this under pressure, sometimes a sysctl vm.drop_caches=3 is required to force the kernel to let go of directory structures, immediately restoring the "missing" memory.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Pro-Tip: If user-space processes and the page cache aren't holding the memory, it is allocated directly within the Kernel Slab Allocator. T."

⚡ 60-Second Elevator Pitch Talking Points

Immediate Triage: If user-space processes and the page cache aren't holding the memory, it is allocated directly
Run targeted verification commands before modifying configuration.
Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.

Q: The OS memory is exhausted, but `top` and `ps` show that the sum of all running applications is only using 20% of the RAM. You clear the page cache, but the memory is still gone. Where is the "missing" memory?

🛠️ Production Runbook & Step-by-Step Resolution

Production Solution & Architecture

📚 Related Production Scenarios in Linux