⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 102 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]

Q: A pod needs GPU resources. How do you configure it?

When a pod needs GPU resources, you configure it by running a device plugin on GPU nodes and requesting the extended resource, for example nvidia.com/gpu: 1, in the container's resources.limits.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>When a pod needs GPU resources, you configure it by running a device plugin on GPU nodes and requesting the extended resource, for example <code>nvidia.com/gpu: 1</code>, in the container's <code>resources.limits</code>.</strong> The scheduler only places the pod on a node that advertises that capacity. Interviewers ask because GPUs are not a built-in resource like CPU: they depend on host drivers, a container runtime hook and a device plugin, and they are expensive, so sharing, taints and cost control matter. The answer should cover the node stack, the pod spec, verification, and the failure modes."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>When a pod needs GPU resources, you configure it by running a device plugin on GPU nodes and requesting the extended resource, for example <code>nvidia.com/gpu: 1</code>, in the container's <code>resources.limits</code>.</strong> The scheduler only places the pod on a node that advertises that capacity. Interviewers ask because GPUs are not a built-in resource like CPU: they depend on host drivers, a container runtime hook and a device plugin, and they are expensive, so sharing, taints and cost control matter. The answer should cover the node stack, the pod spec, verification, and the failure modes.

1️⃣

Mental model: the node stack that exposes a GPU

A GPU appears to Kubernetes only after several layers on the node are healthy.

  • Driver: the vendor driver must be installed on the host, since containers use the host kernel module.
  • Container runtime: NVIDIA Container Toolkit configures containerd or CRI-O so GPU devices and libraries are injected into containers.
  • Device plugin: a DaemonSet registers the extended resource with the kubelet, which then reports nvidia.com/gpu in node capacity and allocatable. AMD and Intel have their own plugins and resource names.
  • Easiest path: the NVIDIA GPU Operator automates driver, toolkit, device plugin and monitoring, while managed clouds often provide GPU node images.
  • Runtime class: some setups require runtimeClassName: nvidia in the pod spec, depending on how the toolkit and containerd were configured. The GPU Operator documents which approach it sets up by default.
2️⃣

Hands-on: configure GPU resources in the pod spec

Request GPUs as an extended resource under limits and keep other requests realistic.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: cuda
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1     # extended resource, integers only
  • Limits are the request: extended resources are specified in limits. If you also set a request, it must equal the limit.
  • Whole numbers only: you cannot request a fraction like 0.5, and GPUs are not overcommitted. Sharing uses time-slicing or MIG configured on the node.
  • Tolerations and selectors: GPU nodes are usually tainted so ordinary pods stay off them, so add a matching toleration and, if needed, a nodeSelector label.
  • Image: use a CUDA-compatible image whose CUDA version is supported by the host driver.
  • Scheduling hints: add node labels from GPU feature discovery, for example the GPU product, so a workload can select an A100 class instead of any GPU. Label names depend on the labeller you deploy.
Advertisement
3️⃣

Verify the GPU is advertised and usable

Confirm at three levels: node capacity, scheduling and inside the container.

kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
kubectl apply -f gpu-test.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/gpu-test --timeout=120s
kubectl logs gpu-test
# Expect the nvidia-smi table listing the GPU model and driver version
  • Node capacity: kubectl describe node should list nvidia.com/gpu under Capacity and Allocatable.
  • Scheduling: a Pending pod with Insufficient nvidia.com/gpu in its events means no node has free GPUs or the toleration is missing.
  • Container: run nvidia-smi in the pod and check that the expected device and driver version appear.
  • Plugin health: check the device plugin pods and their logs if capacity shows zero.
  • Check allocation: kubectl describe node lists allocated extended resources under Allocated resources, which shows how many GPUs are claimed and by which pods.
4️⃣

Failure modes and sharing options

Most GPU incidents are node-stack or scheduling problems rather than pod spec errors.

  • Capacity zero: driver not loaded, toolkit not configured for the runtime, or the plugin DaemonSet not scheduled on the node (taints, selectors).
  • Driver mismatch: CUDA image newer than the host driver supports yields runtime errors, so align versions.
  • Exclusive allocation: one pod per GPU by default, so idle GPUs held by finished or stuck pods waste money. Use Jobs with TTL for batch work.
  • Sharing: NVIDIA MIG partitions supported GPUs into isolated slices, while time-slicing shares a GPU with no memory isolation. Both are configured on the node, not in the pod.
  • Multi-GPU jobs: request several devices in one container, for example nvidia.com/gpu: 4, and remember the scheduler needs all four on one node. Topology and interconnect awareness depend on additional tooling.
Pro Tip: GPU limits must be integers and extended resources cannot be overcommitted, so two pods asking for one GPU each need two physical (or MIG/time-sliced) devices.
5️⃣

Production practice: cost and isolation

GPU nodes are the most expensive part of many clusters, so control who lands on them.

  • Taint GPU nodes: use a taint such as nvidia.com/gpu=present:NoSchedule and only tolerate it in GPU workloads, often enforced with a policy.
  • Quota: apply requests.nvidia.com/gpu ResourceQuota per namespace to cap consumption.
  • Autoscale: scale GPU node groups from zero with the cluster autoscaler or Karpenter, and use priority classes for important jobs.
  • Observe: export GPU metrics (for example NVIDIA DCGM exporter) and alert on utilisation, temperature and ECC errors.
  • Idle detection: compare allocated GPUs with real utilisation from DCGM metrics. Pods that hold a GPU while sitting idle are a common source of waste, so review them regularly and consider automatic cleanup of finished notebooks and jobs.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: A GPU request is only as good as the node stack under it; check node allocatable for nvidia.com/gpu before debugging the pod, and taint GPU nodes with quotas to control cost."
⚡ 60-Second Elevator Pitch Talking Points
  • GPUs are extended resources: the host needs the driver and container toolkit, and a device plugin advertises nvidia.com/gpu to the kubelet.
  • In the pod I set nvidia.com/gpu in limits as an integer, add a toleration for the GPU taint, and verify with node allocatable and nvidia-smi.
  • I automate the stack with the GPU Operator, use quotas, taints and autoscaling for cost, and monitor with DCGM metrics.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Requesting a fractional GPU like 0.5 Extended resources must be integers; sharing needs MIG or time-slicing on the node.
  2. Putting the GPU only under requests Extended resources must be set in limits, and any request must equal the limit.
  3. Forgetting tolerations on tainted GPU nodes The pod stays Pending even though GPUs are free.
  4. Using a CUDA image newer than the host driver The container starts but CUDA calls fail at runtime.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →