⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AI/ML Infrastructure & GPU Interview Questions Scenario 1 of 50 in AI/ML Infrastructure & GPU
Staff AI Infrastructure Engineer AI/ML Infrastructure GPU Orchestration & Kubernetes GPU Orchestration
🎯 Target Role / Context: Staff AI Platform Engineer designing multi-GPU Kubernetes clusters for LLM training and high-throughput inference.

Q: How does the NVIDIA Device Plugin discover, advertise, and allocate GPUs to Kubernetes pods via NVML and Kubelet Device Manager, and how do you configure Kubelet Topology Manager to ensure strict NUMA node alignment between GPUs, CPUs, and high-speed NICs?

Architectural breakdown and troubleshooting of the NVIDIA Kubernetes Device Plugin, NVML communication, Kubelet Topology Manager, and GPU device injection into container runtimes.

#NVIDIA GPU Operator #Device Plugin #NVML #Topology Manager #Kubernetes #AI/ML Infra
🎙️ Candidate Opening & Architectural Context
"Kubernetes core does not natively understand PCI topology or GPU hardware. The NVIDIA Kubernetes Device Plugin interfaces with the NVIDIA Management Library (NVML) on the host and registers extended resources (nvidia.com/gpu) with Kubelet's Device Manager gRPC service. Without proper NUMA and Topology Manager configuration, GPUs on socket 1 may be bound to CPUs on socket 0, causing severe PCIe interconnect contention and throughput penalties."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Kubernetes core does not natively understand PCI topology or GPU hardware. The NVIDIA Kubernetes Device Plugin interfaces with the NVIDIA Management Library (NVML) on the host and registers extended resources (nvidia.com/gpu) with Kubelet's Device Manager gRPC service. Without proper NUMA and Topology Manager configuration, GPUs on socket 1 may be bound to CPUs on socket 0, causing severe PCIe interconnect contention and throughput penalties.

1

Device Discovery, Health Checking, and gRPC Registration

On startup, the NVIDIA Device Plugin daemon pod initializes NVML (nvmlInit), queries all available GPUs (nvmlDeviceGetCount, nvmlDeviceGetHandleByIndex), and opens a gRPC socket at /var/lib/kubelet/device-plugins/nvidia.sock. It calls Register on Kubelet's Registration service, publishing the list of healthy GPU UUIDs as extended resources (nvidia.com/gpu). It streams continuous device health via NVML event listeners.

# Verify Device Plugin registration on node
kubectl describe node ip-10-0-45-12.ec2.internal | grep -A 8 'Allocatable:' | grep 'nvidia.com/gpu'
# Allocatable: nvidia.com/gpu: 8
2

Pod Scheduling and Container Device Allocation Lifecycle

When a pod requests resources (limits: nvidia.com/gpu: 4), the kube-scheduler binds the pod to a node with sufficient unallocated GPU slots. Kubelet's Device Manager executes an Allocate gRPC call to the NVIDIA Device Plugin, passing device IDs. The plugin returns container runtime CDI (Container Device Interface) specs or environment variables (NVIDIA_VISIBLE_DEVICES=GPU-UUID1,GPU-UUID2) which nvidia-container-runtime consumes to mount device nodes (/dev/nvidiaX) and NVML driver libraries into the container.

# Pod manifest requesting GPUs
spec:
  containers:
  - name: llama-inference
    image: vllm/vllm-openai:v0.6.0
    resources:
      limits:
        nvidia.com/gpu: '4'
        memory: 64Gi
        cpu: '32'
Advertisement
3

Enforce Strict NUMA Node Alignment via Kubelet Topology Manager

To prevent cross-socket UPI/QPI traffic bottlenecks during tensor parallel operations, configure Kubelet with static CPU Manager policy and Topology Manager single-numa-node policy. When allocating GPUs, Kubelet coordinates CPU Manager and Device Manager to bind worker pod CPU threads to the exact same NUMA node hosting the PCIe bus for the assigned GPUs and ConnectX InfiniBand/RoCE NICs.

# KubeletConfiguration (/etc/kubernetes/kubelet-config.yaml)
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: 'true'
topologyManagerPolicy: single-numa-node
topologyManagerScope: container
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"The NVIDIA Device Plugin uses NVML to discover and advertise GPU UUIDs to Kubelet via gRPC, and container runtimes inject hardware via CDI. Kubelet Topology Manager with single-numa-node policy is critical to eliminate cross-socket PCIe bandwidth bottlenecks."
⚡ 60-Second Elevator Pitch Talking Points
  • The NVIDIA Device Plugin registers extended GPU resources via gRPC with Kubelet's Device Manager using NVML.
  • When pods request GPUs, nvidia-container-runtime reads CDI device specs to mount driver libraries and character devices without running privileged containers.
  • We configure Kubelet's Topology Manager with single-numa-node policy to co-locate GPUs, CPUs, and high-speed NICs on the same NUMA socket, boosting multi-GPU throughput by up to 25%.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing GPU streaming multiprocessor utilization with raw VRAM video memory allocation when configuring NVIDIA Operator Device A GPU can show 100% VRAM memory allocation while compute cores remain completely idle due to CPU preprocessing bottlenecks or batch starvation.
  2. Running LLM inference endpoints without PagedAttention KV-cache fragmentation controls in vLLM when configuring NVIDIA Operator Device Unmanaged KV cache allocation results in frequent CUDA Out-of-Memory (OOM) errors and limits concurrent request batching capacity.
  3. Employing static batching on inference models instead of continuous dynamic request batching (Triton) when configuring NVIDIA Operator Device Static batching forces requests to wait until batch windows fill, introducing massive latency spikes on variable real-time traffic.
  4. Baking massive multi-gigabyte neural network weights directly inside application container images when configuring NVIDIA Operator Device Baking 50GB weights into image layers creates enormous pod startup times, exhausts node disk space, and blocks rapid scaling.
Advertisement
Want more AI/ML Infrastructure & GPU scenarios?
Explore our complete collection of scenario-based AI/ML Infrastructure & GPU interview runbooks.
Browse All AI/ML Infrastructure & GPU Questions →