Production Server Becomes Slow — Identifying CPU, Memory, Disk, or Network
Mastering the USE Method (Utilization, Saturation, Errors) to isolate system bottlenecks across CPU, Memory, Disk I/O, and Network in under 60 seconds....
Master 89+ battle-tested scenario-based Linux & SRE interview questions for Senior DevOps, Cloud, and SRE engineers. Includes incident runbooks, STAR talking points, and CLI commands.
Mastering the USE Method (Utilization, Saturation, Errors) to isolate system bottlenecks across CPU, Memory, Disk I/O, and Network in under 60 seconds....
Production runbook for recovering from 100% disk utilization: inode exhaustion, du vs df discrepancies, the unlinked open file descriptor trap (lsof deleted), a...
Full command toolkit for diagnosing high CPU processes: thread-level drilldown (top -H), syscall tracing (strace), kernel profiling (perf), and graceful mitigat...
Structured diagnostic sequence for services running in systemd but failing port checks: 127.0.0.1 vs 0.0.0.0 binding, privileged ports, firewall drops, and SELi...
End-to-end SEV-1 incident response lifecycle: automated SLO detection, Incident Command System, immediate mitigation (stopping the bleeding before debugging), d...
Deep diagnostic framework for distinguishing application memory leaks, infinite loops, CFS throttling, external database deadlocks, and traffic anomalies under ...
If ping works but ssh is refused, the server is up (network layer is fine) but the SSH daemon (sshd) is either down, rejecting connection......
This is a classic issue of running out of inodes. While the physical disk blocks might be available, the filesystem has exhausted its all......
First, I'd SSH into the server and run top or htop. I'd check the load average to see the trend (1, 5, 15 minutes)....
Even if the file has chmod +x and the user owns it, execution can be blocked by:...
In Linux, when you rm a file, you delete the directory entry (the link to the inode). However, the disk blocks are not freed as long as a......
In Linux, ports below 1024 are considered "privileged ports." Only processes running as root can bind to them. For security reasons, web ......
777 permissions mean anyone can read, write, and execute the files. Many modern web applications, PHP-FPM, or SSH instances will refuse t......
Intermittent high latency is tricky. My workflow would be:...
If the host has free memory but a process is still OOM killed, the limitation is artificially imposed or architectural:...
The most efficient command would be:...
By default, core dumps are often disabled in production environments due to disk space and security concerns. To enable them:...
To definitively prove if the traffic is leaving the server and if we are receiving a response, I would use tcpdump....
This is almost always an environment variable issue....
Standard rsync -avz over SSH for millions of tiny files is extremely slow. The bottleneck becomes SSH encryption overhead and the serial ......
The D state stands for "Uninterruptible Sleep". The process is waiting deeply within the kernel for a hardware or I/O operation to comple......
First, I verify the block device name (lsblk), format it (mkfs.ext4 /dev/nvmeXn1), and mount it temporarily to verify (mount /dev/nvmeXn1......
If the system becomes completely unresponsive in a hard lockup or OOM stall, manual intervention is slow. I would configure the kernel to......
Since /bin/chmod is essentially just an ELF binary, you can invoke the dynamic linker directly to execute the file without it needing the......
Environment variables are an improvement over hardcoded passwords in git, but they are still insecure because they show up in printenv, /......
An inode (index node) is a data structure on a standard Linux filesystem that stores metadata about a file, like its size, permissions, o......
- Hard Link: A hard link is multiple file paths pointing to the exact same underlying inode (data) on the disk. Because they share the sa......
SysVinit was the legacy approach. It used sequential bash scripts (/etc/init.d/) to start services one by one during boot. It was slow, a......
strace relies on the ptrace system call. Every single time the application tries to do anything, ptrace forces the kernel to pause the ap......
This operator manages file descriptors in Linux: 1 is Standard Output (STDOUT) and 2 is Standard Error (STDERR)....
A Zombie process is a process that has already fully terminated and finished its execution, but still occupies an entry in the process ta......
Linux actively manages memory between Application Memory (Anonymous) and File Caching (Page Cache)....
No, the server is perfectly saturated, but not overloaded....
chroot (Change Root) modifies the root directory (/) for a specific running process and its children. If you chroot an app into /var/jail......
The server is dropping packets because the temporary queue that holds incomplete TCP connections (waiting for the final ACK) is full....
I would use awk. ...
While %util shows the disk is completely saturated (100% time spent doing I/O), it doesn't explain the load profile....
Docker does not store containers as large VM .vmdk files. It uses a Union Filesystem (typically overlay2)....
- su root switches your UID privileges to root, but it explicitly preserves your existing user's environment. You keep your old $PATH, cu......
Since physical access is available, the root password can be reset by interrupting the bootloader (GRUB)....
If user-space processes and the page cache aren't holding the memory, it is allocated directly within the Kernel Slab Allocator. The Slab......
The specialized variable $? holds the numeric exit code of the absolute last command executed in the terminal....
While both achieve the result, ss (Socket Statistics) is vastly preferred and heavily replaces the deprecated netstat....
Yes. This is entirely supported by the /proc and /sys pseudo-filesystems....
Traditional cron is notoriously "dumb." It strictly fires by the clock regardless of the state of the previous job, requiring developers ......
The shell relies on the $PATH environment variable, which is an ordered list of directories (like /usr/bin or /usr/local/bin). When you t......
Modern Linux systems use systemd to manage services instead of SysVinit scripts. A systemd unit file (e.g., /etc/systemd/system/myapp.ser......
The Linux /proc filesystem exposes the kernel's data structures as text files, allowing deep process inspection without external tools....
Namespaces and Cgroups are complementary isolation mechanisms:...
SELinux is a Mandatory Access Control (MAC) layer that sits above traditional Unix permissions (DAC). Even if a file is 644, SELinux can ......
SIGTERM (signal 15): A "polite request" to terminate. The process can catch it, perform cleanup (flush buffers, close connections), and e......
Disk latency has two distinct bottlenecks: read latency (affecting query performance) and write latency (affecting commits/fsync). They r......
Certificate-based SSH (using OpenSSH certificates) eliminates password management and simplifies key rotation across large deployments....
LVM (Logical Volume Manager) allows dynamic volume resizing without unmounting or rebooting....
TCP keepalive is a kernel-level mechanism that sends periodic probe packets on idle connections to verify the remote end is still alive....
When an SSH session disconnects, the shell sends SIGHUP (Hangup signal) to all child processes, which terminates them by default....
Linux uses PAM (Pluggable Authentication Modules) for authentication policy enforcement. PAM configuration files are located in /etc/pam.......
firewalld is the default firewall manager on modern RHEL/CentOS/Fedora systems, replacing direct iptables manipulation. It uses the conce......
systemd-journald stores logs in binary format under /var/log/journal/ (persistent) or /run/log/journal/ (volatile, lost on reboot)....
During a RAID rebuild, the controller must read data from the surviving disks to reconstruct the missing disk's data onto the hot spare. ......
Linux kernel modules must reside in the correct directory and be indexed by depmod before modprobe can find them....
The /etc/hosts file provides static hostname-to-IP mappings. Whether it takes precedence over DNS is controlled by /etc/nsswitch.conf....
Linux uses a nice value (ranging from -20 to +19) to influence CPU scheduling priority. Higher nice values mean lower priority (the proce......
The Linux Audit Framework (auditd) is a kernel-level subsystem that records system calls, file access, and user actions for security comp......
tmpfs is a filesystem that lives entirely in RAM (and optionally Swap). Files stored in tmpfs are never written to physical disk, making ......
Linux maintains a routing table that determines which interface handles traffic to each destination....
xargs reads items from standard input and executes a command with those items as arguments. It is far more efficient than a for loop beca......
The /etc/sudoers file (edited exclusively via visudo to prevent syntax errors) controls which users can run which commands as root....
/etc/passwd contains user account information (username, UID, GID, home directory, shell) and historically stored password hashes. Howeve......
cron is designed for recurring scheduled tasks (e.g., every day at midnight). at is designed for one-time scheduled execution—it runs the......
Linux network bonding combines multiple physical NICs into a single logical interface for redundancy and/or throughput....
The GRUB bootloader controls which kernel is loaded at boot. After a kernel upgrade, the default boot entry may still point to the old ke......
rsyslog is the standard syslog implementation on most Linux distributions. It uses rules in /etc/rsyslog.conf or /etc/rsyslog.d/.conf to ......
The du and df commands measure disk usage differently:...
/dev/null is a special virtual device file known as the "bit bucket" or "black hole." Any data written to it is silently discarded, and r......
POSIX ACLs (Access Control Lists) extend the traditional owner/group/other permission model by allowing per-user and per-group rules on f......
Both screen and tmux are terminal multiplexers that allow you to create persistent terminal sessions that survive SSH disconnections....
Kernel upgrades inherently require a reboot on standard Linux (unlike kpatch/livepatch for security fixes). The goal is to minimize risk ......
An Error Budget is the maximum amount of unreliability permitted over a time window, derived directly from the SLO (Service Level Objecti......
In SRE, this is called Toil—manual, repetitive, automatable, reactive work that scales linearly with service size and adds no enduring va......
SRE postmortems are blameless—they focus on systemic failures, not individual mistakes. The goal is learning and preventing recurrence, n......
A steady, linear increase in memory usage is a classic memory leak pattern. Even if the application functions today, at 1%/day, the serve......
Higher SLOs are exponentially more expensive to achieve, and over-committing creates perverse incentives....
A well-structured runbook is the difference between a 5-minute resolution and a 2-hour scramble. It should be written for the worst case:......
Chaos Engineering is the discipline of proactively injecting controlled failures into production systems to discover weaknesses before th......
On-call is a critical SRE function, but poorly managed on-call leads to burnout, high turnover, and slower incident response....
End-to-end automated pipeline for qualifying, testing, and stress-testing custom Linux AMIs at kernel, driver, storage, and runtime layers before releasing to p...
Diagnostic investigation runbook for isolating sudden p99/p99.9 latency spikes following the deployment of a sidecar-based caching proxy (e.g. Redis/Memcached s...
Mastering system performance diagnostics: differentiating compute-bound CPU saturation from disk/network I/O bottlenecks using load averages, %wa iowait, top, v...