Interview Experience at Nvidia – Senior DevOps Engineer (GPU Infrastructure & Bare Metal)

Order of Execution & Diagnostic Flow:

1. BMC / IPMI Out-of-Band: Connect to BMC via Redfish/IPMI. Inspect System Event Log (ipmitool sel elist). Verify power rails, CPU/DIMM initialization, and that all 8 GPU sockets pass hardware power-on self tests (POST).
2. Firmware / UEFI: Verify UEFI boot sequence and ensure PXE NIC is enabled. Confirm Secure Boot keys match kernel signatures and PCIe Option ROMs for Mellanox/NVIDIA NICs are initialized.
3. PXE / DHCP / iPXE: Tail DHCP logs on infrastructure server for DHCPDISCOVER, DHCPOFFER, and DHCPACK. Verify Option 66 (Next-Server/TFTP IP) and Option 67 (bootfile name like ipxe.efi). Verify TFTP/HTTP transfer of vmlinuz and initramfs.
4. Bootloader (GRUB2 / iPXE): Check kernel command-line arguments: verify serial console redirection (console=tty0 console=ttyS0,115200), root device identifier (root=UUID=...), and GPU flags (intel_iommu=on iommu=pt).
5. Initramfs / OS Kernel: Kernel decompresses initramfs and loads storage controllers (NVMe) and network drivers (mlx5_core). Check for driver panic, missing root filesystem mount, or pivot_root failure into dracut emergency shell.
6. Systemd & Kubelet Registration: Verify systemd-networkd assigns management IP. Verify containerd is active. Check Kubelet logs (journalctl -u kubelet -e --no-pager) for TLS bootstrap token validation, cluster API server reachability (port 6443), and pending CSR approval via kubectl get csr.

# Triage Sequence CLI Commands
ipmitool -I lanplus -H <bmc_ip> -U admin -P secret sel elist | tail -n 25
journalctl -u kubelet -n 100 --no-pager | grep -iE 'error|tls|bootstrap|failed'
kubectl get nodes -o wide && kubectl get csr

Non-Reboot Live Diagnostic Procedure:

• 1. Inspect Driver & PCIe Health (AER & Xid Errors): Check kernel messages for NVIDIA driver Xid errors: dmesg -T | grep -iE 'nvidia|aer|xid'. Xid 31/32 indicates GPU memory page fault; Xid 79 indicates the GPU physically fell off the PCIe bus (link drop due to power clamp or PCIe signal integrity). Inspect PCIe Advanced Error Reporting via lspci -vvv -s <pci_addr> | grep -E 'UESta|CESta'. Spiking uncorrectable errors under load proves hardware/PCIe bus degradation.
• 2. Verify IOMMU Group Isolation: Inspect /sys/kernel/iommu_groups/*/devices/ for the target GPU's group. If the GPU shares its IOMMU group with another host device (e.g., SATA controller or PCIe bridge), any reset or power state change in that sibling device tears down the entire IOMMU group, dropping the passthrough container. Check dmesg | grep -i dmar for DMA remap faults.
• 3. Workload Memory Pinning & Locked Limits: GPU Direct and VFIO require host physical memory to be locked (mlock) to prevent swapping during DMA operations. Check the container process limits: grep -i locked /proc/<pid>/limits (Max locked memory). If LimitMEMLOCK is not set to infinity or if vm.nr_hugepages is exhausted under burst allocations, cudaMallocHost() fails, aborting DMA transfers without crashing the OS.

# Check PCIe AER and IOMMU group mapping live
lspci -tv | grep -i nvidia
for d in /sys/kernel/iommu_groups/*/devices/*; do echo -e "${d%%/devices*}\t${d##*/}"; done | grep <pci_id>
cat /proc/<workload_pid>/status | grep -i vmlck

Root Causes & Proof Techniques:

• Cause 1: Inode Exhaustion (The Missing Inodes): df -h measures block usage (bytes). ML datasets with millions of tiny checkpoint files or temporary shards exhaust filesystem inodes while leaving disk capacity at 40%.
  Proof: Run df -i. If IUse% is 100%, Kubelet triggers DiskPressure eviction immediately.
• Cause 2: Unlinked Deleted Files Held by Active PIDs: When log files or temporary datasets are unlinked (rm large.log) while a running process (containerd, Python training script, or Fluentd) holds the file descriptor open, directory entries disappear (invisible to du) but disk blocks remain locked.
  Proof: Run lsof +L1 or lsof | grep -i deleted. Truncate the file descriptor live: : > /proc/<pid>/fd/<fd_num>.
• Cause 3: Files Hidden Beneath Mount Overlays: Data written to /var/lib/containerd before an external NVMe disk was mounted on top of it consumes root filesystem blocks that are masked by the mount.
  Proof: Run mount --bind / /mnt/root and inspect with du -sh /mnt/root/var/lib/containerd.
• Cause 4: Kubelet Separate Storage Thresholds (nodefs vs imagefs): Kubelet monitors root filesystem (nodefs) and container storage (imagefs) independently. If container layers on /var/lib/containerd hit 85% on a dedicated partition, Kubelet fires DiskPressure even if root / is 40%.

# Prove inodes and deleted open file descriptors
df -i / && df -h /
lsof +L1 | sort -nk 7 -r | head -n 10
crictl info | jq '.status.storage'

Remote Recovery Sequence:

1. Activate Serial-over-LAN (SoL) & KVM: Open remote console via ipmitool -I lanplus -H <bmc_ip> -U admin -P pass sol activate. Record the exact POST checkpoint code (e.g. 0x55 Memory Initialization, 0x92 PCI Training).
2. Analyze BMC System Event Log: Run ipmitool sel elist. Check for power rail fault, PCIe slot training failure, or thermal safety trip.
3. Out-of-Band BIOS Flashback via Redfish: Power down the host CPU (ipmitool power off). Use the BMC Redfish API or web console to flash the known-stable "Golden" BIOS binary directly into the primary SPI flash chip while the host is powered down.
4. Toggle Dual-BIOS Bank: Modern enterprise boards feature dual SPI flash ROMs. Send an OEM IPMI raw command or Redfish action (Bios.ResetBios) to flip the hardware multiplexer to boot from the backup BIOS ROM.
5. Clear CMOS & NVRAM via BMC: Corrupted NVRAM variables cause 70% of POST hangs after major firmware updates. Issue an IPMI command to clear CMOS/NVRAM to factory defaults, followed by an AC power cycle (ipmitool chassis power cycle) allowing capacitors to drain for 30s.

Architecture, Scheduling & What Breaks:

• NUMA Topology: Dual-socket servers (e.g. AMD EPYC / Intel Xeon) split physical cores, RAM, and PCIe root complexes across NUMA nodes. Sockets communicate over inter-socket interconnects (AMD Infinity Fabric / Intel UPI) with higher latency and lower bandwidth than local memory buses.
• Kernel Scheduling: Linux CFS allocates memory on the local NUMA node first. When kernel.numa_balancing=1 is enabled, the kernel unmaps pages to trigger page faults, tracking if memory should migrate to follow a thread.
• What Breaks in GPU Training Jobs:
  • PCIe & Cross-Socket Bottleneck: If PyTorch DataLoader threads run on CPU cores on NUMA Node 1 while the target GPU is physically attached to PCIe switches on NUMA Node 0, every tensor batch traverses the UPI link. This throttles Host-to-Device (H2D) throughput from 64 GB/s down to 18 GB/s, leaving GPU compute utilization hovering at 25%.
  • GPUDirect RDMA Breakdown: ConnectX-7 InfiniBand NICs perform direct DMA to GPU VRAM over PCIe. If the NIC is on NUMA 0 and the GPU is on NUMA 1, GPUDirect RDMA fails or incurs cross-socket penalties, degrading NCCL AllReduce distributed training throughput by 50%+.
  • Page Fault Stalls: Kernel automatic NUMA balancing induces periodic page table unmapping, injecting 50-100ms micro-stalls into model training steps.

# Enforce strict NUMA affinity for GPU workloads
numactl --cpunodebind=0 --membind=0 python train.py
nvidia-smi topo -m   # Inspect GPU, NVLink, and NUMA node affinity

Finding the 36G Discrepancy:

• 1. TSO / Segmentation Offload Miscalculation: iftop uses libpcap to capture raw packets at the Linux socket layer before hardware segmentation offload (TSO/GSO). The host calculates 40G based on 64KB aggregated virtual packets, but if the MTU on the switch port is 1500 while the host MTU is 9000, the NIC or switch drops the oversized packets, leaving only 4G of surviving wire traffic.
  Verify: ethtool -k <interface> | grep -i segment and check MTU alignment.
• 2. Lossless RoCE / PFC (Priority Flow Control) PAUSE Frames: In GPU clusters running RoCEv2, congested downstream switches emit 802.1Qbb PFC PAUSE frames. The NIC stops transmitting while the Linux kernel continues buffering and queuing data.
  Verify: Run ethtool -S <interface> | grep -iE 'pause|pfc|discard'.
• 3. LACP / Bond Hash Polarization: If the host uses an 802.3ad port-channel (bond0) across dual 25G/40G interfaces, the hashing policy (xmit_hash_policy=layer2 vs layer3+4) may funnel 90% of traffic down an alternate slave port not being measured by the single switch interface monitor.
  Verify: Run cat /proc/net/bonding/bond0 and inspect traffic distribution across all physical slaves.
• 4. Physical Layer Bit Errors & Optical Signal Degradation: Check for CRC/FCS frame errors and optical dBm levels: ethtool -m <interface> and ethtool -S <interface> | grep -iE 'crc|error|drop'.

Zero-Downtime Reconciliation Flow:

1. Refresh State Without Applying: Run terraform plan -refresh-only to synchronize the remote state file with actual cloud reality without altering infrastructure or code. Export the plan to JSON (terraform show -json) to inspect the exact drift diff.
2. Triage Drift (In-Place vs Destructive Replacement): Separate changes into safe attributes (tags, security group egress rules, scaling parameters) vs destructive attributes (subnet changes, KMS keys, instance types requiring termination).
3. Codify Necessary Operational Fixes: If the console changes were emergency hotfixes that must persist, update the HCL code locally to reflect the manual console state. Re-run terraform plan until the diff reports No changes. Your infrastructure matches the configuration.
4. State Adoption for Re-created Objects: If an operator recreated a resource out-of-band with a new resource ID, use terraform state rm <resource_address> followed by terraform import <resource_address> <new_cloud_id> to bind the new entity into state without destroying live workloads. Use lifecycle { ignore_changes = [...] } for attributes legitimately managed by autoscalers or external operators.

Top 3 Causes & Isolation Techniques:

• 1. Missing State from Uncreated Prerequisite Artifacts: In --check mode, tasks like file: state=directory or git: clone report "changed" but do not create the actual directory or files on disk. Subsequent tasks that rely on those files (e.g. copy: dest=/dir/conf or command executions) pass check mode because downstream tasks are skipped or simulated, but fail immediately in live execution when the path is missing.
  Isolation: Look for tasks using register: result where downstream tasks evaluate result.stat.exists.
• 2. Shell / Command Tasks Bypassing Check Mode: Raw commands (command:, shell:) do not execute during --check unless explicitly annotated with check_mode: no. During real execution, these shell commands run, hitting syntax errors, missing environment variables (e.g. PATH, CUDA_HOME), or non-zero exit codes.
  Isolation: Audit playbooks for shell: and command:. Replace them with idempotent native modules.
• 3. Dynamic Privilege Escalation & Environmental Lock Contention: In --check, Ansible simulates package state checks. In live runs, executing apt or yum hits package manager locks held by unattended-upgrades (/var/lib/dpkg/lock-frontend) or SELinux context denials when writing to system directories.
  Isolation: Run ansible-playbook -vvv --step to execute tasks interactively step-by-step and inspect Python tracebacks in ~/.ansible/tmp/.

Enterprise Ring Architecture & Rollout Plan:

• Ring 0 (Canary 1% - 2 nodes, 24h bake): Select 1 node per rack/failure domain. Run automated qualification: CUDA GEMM compute benchmarks, GPUDirect RDMA bandwidth tests, and Kubernetes pod evictions.
• Ring 1 (Canary 5% - 10 nodes, 12h bake): Schedule real production training batch jobs. Monitor dmesg, PCIe AER counters, and NCCL AllReduce latency.
• Ring 2 (Broad Batch 25% - 50 nodes): Upgrade nodes in batches of 10 within one rack domain at a time to prevent cross-rack partitioning.
• Ring 3 (Full Fleet Remaining 70% - 138 nodes): Rolling upgrades across remaining nodes with a maximum concurrency limit of 10 draining nodes.
• Drain & Reboot Automation: kubectl cordon <node> → kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force → install kernel package via Ansible → reboot via kexec (or IPMI) → post-boot validation script (verify uname -r, nvidia-smi, ibv_devinfo) → kubectl uncordon <node>.
• Hard Rollback Criteria:
  1. Any node encounters kernel panic, GPU Xid error, or PCIe uncorrectable error.
  2. Distributed training NCCL AllReduce latency degrades by > 5%.
  3. Pod eviction / drain times out > 15 minutes due to PDB deadlock.
  4. Rollback Action: Pause deployment pipeline, set GRUB default to fallback kernel (grub2-set-default 1 && reboot), and uncordon.

Self-Proving Artifact Guardrails:

• 1. Hermetic Sandbox Integration Stage (Testcontainers): Do not trust clean exit codes from compiler stages. Boot the artifact inside an isolated Docker sandbox with real database/service dependencies and execute an automated integration test suite against the live binary.
• 2. Container Runtime Smoke & Linker Verification: Inspect binary dependencies inside the container image using ldd to guarantee no dynamic shared libraries (.so) are missing. Execute an in-container entrypoint test verifying that CUDA runtime libraries match host driver API capabilities (nvidia-smi compatibility check).
• 3. Cryptographic SBOM & Vulnerability Gating: Generate a Software Bill of Materials with Syft. Scan the image with Trivy/Grype, and block promotion if High/Critical CVEs exist. Sign verified artifacts with Cosign/Sigstore before pushing to the production registry.
• 4. Automated Canary Analysis (ACA): Deploy newly built artifacts to a 2% traffic canary slice evaluated by Prometheus for 15 minutes. Automatically verify error rate < 0.01% and P99 latency within 5% of baseline before allowing production deployment.

Liabilities & Architecture Tradeoffs:

• Ansible Module as a Liability: Modules are standalone Python execution units. Writing custom in-house Python modules becomes a liability when developers reinvent capabilities already provided by community built-ins. Custom modules lack upstream security maintenance, require maintaining Python virtual environments and dependencies on all target hosts, and break silently during Python minor version upgrades across the cluster.
• Ansible Role as a Liability (Role Bloat): Roles bundle tasks, handlers, variables, and templates. A role becomes a liability when teams build monolithic mega-roles that configure too many unrelated services (e.g. GPU drivers, Docker, networking, and user management in a single role). Deep variable inheritance hierarchies (group_vars vs host_vars vs defaults/main.yml) create hidden state collisions, making roles impossible to test in isolation and causing a bug in one task to block unrelated cluster maintenance.

Forensic Diagnostic Walkthrough:

1. Inspect K8s Events: Run kubectl describe pod <pod_name>. Inspect the Events: section for exact error strings: 401 Unauthorized, 403 Forbidden, dial tcp: i/o timeout, or no space left on device.
2. Registry Authentication & Token Expiry: If 401/403, verify imagePullSecrets in the pod spec or default ServiceAccount. Check if cloud IAM tokens (AWS ECR / GCP Artifact Registry / Harbor robot accounts) expired or lack repository read permissions. Test pull directly on the node: crictl pull --creds <user>:<token> <image_url>.
3. Network Policy & Node Egress: If connection times out, test DNS resolution from the node: dig @10.96.0.10 registry.internal.company.com. Check Calico/Cilium egress network policies to confirm outbound traffic to port 443 is permitted. Inspect containerd proxy configuration in /etc/containerd/config.toml (missing NO_PROXY for internal registries).
4. Image Digest & Architecture Mismatch: Verify image digest architecture. If the image was built for linux/arm64 and deployed to an x86_64 GPU node, containerd pulls the manifest list but fails during image unpack with exec format error or no matching manifest for linux/amd64.
5. Node Disk Pressure: If the node's container storage filesystem (/var/lib/containerd) is full, containerd halts layer downloads with write /var/lib/containerd/...: no space left on device. Clean dangling layers with crictl rmi --prune.

The Forensic Reality:

• 1. Inode Exhaustion on Container Runtime Filesystem: Machine learning distributed training runs generate millions of tiny temporary checkpoint shards, token files, or IPC sockets. df -h only checks byte capacity (showing 30%), but df -i /var/lib/containerd is at 100%. When free inodes drop below Kubelet's eviction threshold (imagefs.inodesFree < 5%), Kubelet flags DiskPressure and evicts active pods.
• 2. Separate Mount for Container Storage: The root filesystem (/) is at 30%, but /var/lib/containerd resides on a dedicated NVMe partition or LVM volume that has crossed the 85% eviction threshold. Kubelet monitors imagefs separately from nodefs.
• 3. Zombie Overlay2 Layers Locked by Crashed Processes: Desynchronization between containerd and the Linux kernel can leave orphaned overlay2 directories uncollected, consuming storage hidden from standard directory traversals.

# Verify inodes and Kubelet eviction thresholds
df -i /var/lib/containerd
journalctl -u kubelet -e --no-pager | grep -i eviction
crictl stats

CI vs Canary Reality & Proof Flow:

• Why It Happens: CI environments run single-threaded synthetic tests with tiny mock data where memory limits are rarely constrained. In production Canary, the container receives real traffic concurrency, live payload sizes, and strict resources.limits.memory.
• Specific GPU / ML Memory Leak Causes:
  • PyTorch DataLoader Pinned Memory: pin_memory=True allocates page-locked host RAM for asynchronous GPU DMA transfers. If batch queues pile up faster than GPUs consume them, pinned memory balloons linearly until the cgroup limit kills the process.
  • Shared Memory (/dev/shm) Exhaustion: PyTorch multiprocessing workers share tensor data via /dev/shm. If the pod manifest does not mount an emptyDir with medium: Memory to /dev/shm, the default 64MB container shared memory fills up and crashes.
• How to Prove It:
  1. Inspect kernel dmesg: dmesg -T | grep -iE 'oom_reaper|killed process'. Identify the exact process name, RSS memory, and cgroup path.
  2. Check cgroup v2 memory events: cat /sys/fs/cgroup/system.slice/.../memory.events to view the oom_kill counter.
  3. Inspect Prometheus container_memory_working_set_bytes graph over the 90-second ramp-up to observe the memory consumption trajectory.

The Real-World Cascading Outage Scenario:

• During a traffic surge, the downstream PostgreSQL database experienced connection pool saturation, increasing query response latency from 3ms to 3,500ms.
• The Liveness probe was configured with timeoutSeconds: 1 and failureThreshold: 3. Within 15 seconds, all 100 pods in the cluster failed their liveness probes simultaneously.
• Kubelet abruptly restarted every single pod in the cluster. As the 100 pods booted simultaneously, their startup routines flooded the struggling database with 100 new connection handshakes, keeping database CPU pegged at 100%.
• Upon boot, their liveness probes immediately failed again. The cluster was trapped in an infinite cascading restart loop for 2 hours until operators manually disabled the liveness probes.

Pre-Rollback Diagnostic Order:

1. Pod State & Events: Run kubectl get pods -l app=service and kubectl describe pod <stuck_pod>. Determine if new replicas are in Pending, CrashLoopBackOff, or Running (0/1 Ready).
2. Cluster Capacity & GPU Resource Exhaustion: If pods are Pending, check Events: FailedScheduling. Did the new replicas request dedicated GPU resources (nvidia.com/gpu: 8) that exceed available allocatable GPUs on the cluster?
3. Readiness Probe Failures: If pods are Running but not Ready, inspect container logs (kubectl logs <stuck_pod>). The deployment controller will not terminate old pods until new pods pass their readiness checks.
4. PodDisruptionBudget (PDB) Constraints: Check if a strict PDB (minAvailable: 80%) prevents Kubernetes from evicting old replicas.
5. Deployment Strategy Constraints: Check maxSurge and maxUnavailable in the deployment spec. If maxSurge: 0 and maxUnavailable: 0, the deployment controller is mathematically deadlocked from creating new pods or terminating old ones.
6. PersistentVolume Access Modes: If pods require ReadWriteOnce (RWO) PersistentVolumes, check if old pods are still holding the storage lock, preventing new pods on different nodes from mounting the volume.

First 30 Days Strategic Roadmap:

• Days 1–10: Discovery, Incident Analysis & Operational Friction Mapping:
  • Audit past 6 months of P1/P2 tickets, on-call paging logs, and Slack incident channels.
  • Identify the 20% of repetitive operational tasks that consume 80% of on-call engineer hours (e.g. manual node draining, disk cleanup, GPU driver reinstall, certificate renewals).
  • Construct an Automation Prioritization Matrix evaluated by: Frequency × Risk × Engineering Effort.
• Days 11–20: Codify Tribal Knowledge Before Automating:
  • Automating unwritten, chaotic processes produces automated chaos.
  • Pair with the engineers holding the tribal knowledge to convert mental routines into standardized Markdown runbooks with explicit input parameters, diagnostic commands, and rollback procedures.
• Days 21–30: Ship High-Impact Quick Wins & Reset Expectations:
  • Automate 2 high-frequency, low-risk operational workflows (e.g. automated unhealthy node cordon/drain, stale overlay2 garbage collection).
  • Implement them as modular code with pull requests, linting, and human-in-the-loop verification gates.
  • Present quantifiable MTTR reduction metrics to executive leadership to reset "automate everything" into an achievable, phased milestone roadmap.

Architectural Pushback & Real Cost Breakdown:

• 1. Explosive Blast Radius: A single typo, syntax error, or breaking shared library change in the monolithic pipeline script blocks deployments across the entire engineering organization simultaneously.
• 2. Build Queue Starvation & Velocity Destruction: When dozens of teams push commits across different services, the monolithic pipeline triggers redundant build stages, test suites, and linting passes. CI queue wait times balloon from 5 minutes to 45+ minutes, destroying deployment frequency and developer feedback loops.
• 3. Security Principle of Least Privilege Violation: A single pipeline requires broad administrative credentials (access to all production Kubernetes clusters, AWS IAM roles, and package registries). If a junior service needs to deploy, it runs with root-level cluster credentials.
• 4. Cache Thrashing & Agent Saturation: Caching container layers and language dependencies across 15 disparate tech stacks in a single pipeline leads to constant cache eviction and disk bloat on build agents.
• Recommended Alternative: Decentralized, modular pipeline architecture using standardized reusable templates (Jenkins Shared Libraries or GitHub Actions Composite Actions). Each microservice maintains its isolated pipeline config triggering only on its own path changes (paths: ['services/gpu-scheduler/**']).

Blameless SRE Postmortem Document: