Linux Internals & SRE Incident Interview Questions (2026 Edition)
⚡ 135 Live Scenarios
🎯 STAR Method Answers📋 60s Elevator Pitches
Linux operating system internals and Site Reliability Engineering (SRE) principles form the bedrock of scalable internet infrastructure, demanding thorough operational command from senior infrastructure professionals. Modern engineering leadership expects SRE candidates to navigate ambiguous production outages with calm, structured troubleshooting methodologies rooted in the USE (Utilization, Saturation, and Errors) and RED (Rate, Errors, and Duration) frameworks. Technical interviewers evaluate your capability to dissect system load spikes using tools like top, vmstat, iostat, and pidstat, track kernel thread locks using dmesg, and trace hung system calls using strace and eBPF BCC tools. You are expected to design realistic Service Level Objectives (SLOs) and Error Budgets that effectively align feature delivery velocity with enterprise uptime commitments. Review our scenario-driven Linux and SRE interview questions to master incident post-mortem retrospectives, blameless RCA documentation, systemd daemon watchdog architectures, out-of-memory OOM-killer tuning, and kernel sysctl parameter tuning for ultra-high-throughput networking stacks operating under heavy enterprise concurrency.
Get 1 Linux & SRE interview question in your inbox every week
Preparing for a cloud or platform interview? Subscribe to get our weekly deep-dive scenario plus instant access to the Top 50 Kubernetes Interview Questions & Incident Runbooks PDF.
Mastering the USE Method (Utilization, Saturation, Errors) to isolate system bottlenecks across CPU, Memory, Disk I/O, and Network in under 60 seconds....
Production runbook for recovering from 100% disk utilization: inode exhaustion, du vs df discrepancies, the unlinked open file descriptor trap (lsof deleted), a...
Full command toolkit for diagnosing high CPU processes: thread-level drilldown (top -H), syscall tracing (strace), kernel profiling (perf), and graceful mitigat...
Structured diagnostic sequence for services running in systemd but failing port checks: 127.0.0.1 vs 0.0.0.0 binding, privileged ports, firewall drops, and SELi...
Deep diagnostic framework for distinguishing application memory leaks, infinite loops, CFS throttling, external database deadlocks, and traffic anomalies under ...
A regional failover is the most destructive action an SRE can take; it risks massive data loss, split-brain scenarios, and hours of complex resolution. It shoul...
Never SSH and patch: If an EC2 instance needs a security update or a configuration change, you never SSH into the live server to run apt-get upgrade or edit a c...
When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: Integration Dat...
Continuous Integration (CI): Developers frequently merge their code changes into a central repository mainline. Automated builds and unit tests run immediately ...
In microservices, Service A often calls Service B. If Service B becomes unresponsive, Service A will continuously wait for timeouts, backing up its own queues a...
You cannot simply run an ALTER TABLE RENAME command, as the live application will instantly crash when it queries the old name. The solution is the Expand and C...
Chaos Engineering is the practice of systematically injecting controlled failures (like terminating instances, simulating packet loss, or degrading database res...
In distributed architectures (where data is replicated across multiple servers or regions for high availability), it takes time for a write on Node A to propaga...
Lift and Shift (Rehosting) means cloning the physical/virtual machines byte-for-byte directly into identical cloud VMs (like AWS EC2) with zero architectural ch...
Forcing the client browser or mobile app to manage complex network orchestration, aggregations, and individual microservice latencies creates a horrible user ex...
If ping works but ssh is refused, the server is up (network layer is fine) but the SSH daemon (sshd) is either down, rejecting connections (e.g., max connection...
This is a classic issue of running out of inodes. While the physical disk blocks might be available, the filesystem has exhausted its allocation of inodes, whic...
In Linux, when you rm a file, you delete the directory entry (the link to the inode). However, the disk blocks are not freed as long as any running process hold...
In Linux, ports below 1024 are considered "privileged ports." Only processes running as root can bind to them. For security reasons, web servers and a...
permissions mean anyone can read, write, and execute the files. Many modern web applications, PHP-FPM, or SSH instances will refuse to read configurations or ex...
Standard rsync -avz over SSH for millions of tiny files is extremely slow. The bottleneck becomes SSH encryption overhead and the serial nature of rsync analyzi...
The D state stands for "Uninterruptible Sleep". The process is waiting deeply within the kernel for a hardware or I/O operation to complete (often a h...
If the system becomes completely unresponsive in a hard lockup or OOM stall, manual intervention is slow. I would configure the kernel to panic and automaticall...
Since /bin/chmod is essentially just an ELF binary, you can invoke the dynamic linker directly to execute the file without it needing the execute bit. The dynam...
Environment variables are an improvement over hardcoded passwords in git, but they are still insecure because they show up in printenv, /proc//environ, and cras...
An inode (index node) is a data structure on a standard Linux filesystem that stores metadata about a file, like its size, permissions, ownership, and pointers ...
Hard Link: A hard link is multiple file paths pointing to the exact same underlying inode (data) on the disk. Because they share the same inode, if you delete t...
SysVinit was the legacy approach. It used sequential bash scripts (/etc/init.d/) to start services one by one during boot. It was slow, and writing robust bash ...
We encountered this OS-level bottleneck during peak traffic and diagnosed it down to kernel and filesystem metrics. The interviewer is testing: The ptrace overh...
chroot (Change Root) modifies the root directory (/) for a specific running process and its children. If you chroot an app into /var/jail, the app believes /var...
su root switches your UID privileges to root, but it explicitly preserves your existing user's environment. You keep your old $PATH, current working direct...
Traditional cron is notoriously "dumb." It strictly fires by the clock regardless of the state of the previous job, requiring developers to write comp...
The shell relies on the $PATH environment variable, which is an ordered list of directories (like /usr/bin or /usr/local/bin). When you type a command, the syst...
Modern Linux systems use systemd to manage services instead of SysVinit scripts. A systemd unit file (e.g., /etc/systemd/system/myapp.service) must configure re...
SELinux is a Mandatory Access Control (MAC) layer that sits *above* traditional Unix permissions (DAC). Even if a file is 644, SELinux can block access if the p...
SIGTERM (signal 15): A "polite request" to terminate. The process can catch it, perform cleanup (flush buffers, close connections), and exit cleanly....
Linux uses PAM (Pluggable Authentication Modules) for authentication policy enforcement. PAM configuration files are located in /etc/pam.d/ and control how auth...
firewalld is the default firewall manager on modern RHEL/CentOS/Fedora systems, replacing direct iptables manipulation. It uses the concept of zones (groups of ...
Linux uses a nice value (ranging from -20 to +19) to influence CPU scheduling priority. Higher nice values mean lower priority (the process is "nicer"...
The Linux Audit Framework (auditd) is a kernel-level subsystem that records system calls, file access, and user actions for security compliance and forensics....
tmpfs is a filesystem that lives entirely in RAM (and optionally Swap). Files stored in tmpfs are never written to physical disk, making read/write operations e...
/etc/passwd contains user account information (username, UID, GID, home directory, shell) and historically stored password hashes. However, /etc/passwd must be ...
rsyslog is the standard syslog implementation on most Linux distributions. It uses rules in /etc/rsyslog.conf or /etc/rsyslog.d/*.conf to route log messages bas...
Complete guide to /dev/null in Linux: understanding the black hole pseudo-device, stdout and stderr suppression, benchmarking disk writes, and creating empty fi...
POSIX ACLs (Access Control Lists) extend the traditional owner/group/other permission model by allowing per-user and per-group rules on files and directories....
Kernel upgrades inherently require a reboot on standard Linux (unlike kpatch/livepatch for security fixes). The goal is to minimize risk and ensure rollback cap...
SRE postmortems are blameless—they focus on systemic failures, not individual mistakes. The goal is learning and preventing recurrence, not assigning blame....
A steady, linear increase in memory usage is a classic memory leak pattern. Even if the application functions today, at 1%/day, the server will OOM in approxima...
Chaos Engineering is the discipline of proactively injecting controlled failures into production systems to discover weaknesses before they cause real outages....
End-to-end automated pipeline for qualifying, testing, and stress-testing custom Linux AMIs at kernel, driver, storage, and runtime layers before releasing to p...
Diagnostic investigation runbook for isolating sudden p99/p99.9 latency spikes following the deployment of a sidecar-based caching proxy (e.g. Redis/Memcached s...
Strategic framework for transitioning an engineering organization from ignored vanity SLO dashboards to legally binding Error Budgets that govern deployment vel...
High-urgency architecture strategy to deliver true multi-region automated traffic failover in 3 weeks, bypassing the DNS TTL caching bottleneck using Anycast BG...
Executive communication framework for translating technical refactoring, platform engineering, and cloud modernization projects into hard financial ROI and reve...
Mastering system performance diagnostics: differentiating compute-bound CPU saturation from disk/network I/O bottlenecks using load averages, %wa iowait, top, v...
Step-by-step diagnostic workflow for triaging a 15x response time regression (200ms to 3s) immediately following a production deployment without catastrophic er...
How to answer 'Tell me about a real production incident you handled and what was the root cause' using the STAR method (Situation, Task, Action, Resul...
High-stakes SRE decision framework when an automated rollback script fails during a production outage with only 5 minutes remaining before an enterprise SLA bre...
Master-level behavioral and technical narrative answering the classic senior interview question: 'Tell me about a time you broke production, what you learn...
Troubleshooting checklist when password-based SSH authentication fails to function despite editing /etc/ssh/sshd_config to set PasswordAuthentication yes....
How to operate effectively in a 24/7 rotational on-call support model, establish clear boundaries between Operations and Backend teams, and prevent alert fatigu...
Key incident runbooks, interview talking points, and architecture tradeoffs.
Production server becomes slow — how do you identify CPU/memory/disk/network issues?
When a production server becomes slow, I apply Brendan Gregg's USE Method (Utilization, Saturation, Errors) using a systematic 60-second command checklist to isolate which subsystem is bottlenecking.
Key Architectural Takeaway: Use the USE Method. Run 'uptime' -> 'vmstat 1 5' -> 'free -h' -> 'iostat -xz 1 5'. High 'wa' in vmstat means Disk I/O bottleneck; high 'si/so' means memory swapping; high 'r' means CPU saturation.
Disk reaches 100% — how would you find and safely remove the cause?
When a disk hits 100%, services begin dropping connections, crash, and cannot write logs or create PID files. The investigation requires checking both disk blocks and inodes, and avoiding the classic 'deleted file still held open' trap.
Key Architectural Takeaway: Check 'df -i' for inode exhaustion. Use 'lsof +L1' to find deleted files held open by running processes, and truncate them via /proc/<PID>/fd/<FD> rather than rebooting.
Process is consuming high CPU — which commands would you use?
When a process consumes high CPU, my goal is to drill down from the system process level down to the exact thread, system call, or application function causing the burn.
Key Architectural Takeaway: Use 'top -c' to find PID, 'top -H -p <PID>' to find the specific thread, and 'strace -c' or 'perf top' to pinpoint the looping function or lock contention.
Service is running but not listening on the expected port — how would you troubleshoot?
When systemctl reports 'active (running)' but you cannot connect to the expected port, I trace through 5 discrete checkpoints: daemon listening sockets, IP binding address, privileged port permissions, firewall rules, and security modules.
Key Architectural Takeaway: Check 'ss -tulpn | grep <PID>' first. If listening on 127.0.0.1, change binding to 0.0.0.0. If port < 1024, check CAP_NET_BIND_SERVICE. If bound correctly, check iptables and SELinux port policies.
A production deployment causes a sudden 40% increase in error rates. Walk through your incident-response process from detection and diagnosis to rollback, mitigation, root-cause analysis, and post-incident prevention.
A 40% error rate spike is a SEV-1 outage. My golden rule as an Incident Commander is: 'Mitigate first to protect customers; debug second.' Never spend hours debugging a broken deployment in production when an immediate rollback restores service.
Key Architectural Takeaway: Mitigate first, debug second. Stop customer bleeding via immediate rollback or feature flag disablement within 5 minutes. Formally separate Incident Commander, Ops Lead, and Comms Lead, followed by a blameless post-mortem.
🌐
Explore Related DevOps & Cloud Domains
Cross-train across interconnected systems for senior and staff infrastructure rounds.
Master Linux & SRE & Platform Engineering In Production
Preparing for a cloud or platform interview? Subscribe to get our weekly deep-dive scenario plus instant access to the Top 50 Kubernetes Interview Questions & Incident Runbooks PDF.