Q: A pod needs GPU resources. How do you configure it?
When a pod needs GPU resources, you configure it by running a device plugin on GPU nodes and requesting the extended resource, for example nvidia.com/gpu: 1, in the container's resources.limits.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>When a pod needs GPU resources, you configure it by running a device plugin on GPU nodes and requesting the extended resource, for example <code>nvidia.com/gpu: 1</code>, in the container's <code>resources.limits</code>.</strong> The scheduler only places the pod on a node that advertises that capacity. Interviewers ask because GPUs are not a built-in resource like CPU: they depend on host drivers, a container runtime hook and a device plugin, and they are expensive, so sharing, taints and cost control matter. The answer should cover the node stack, the pod spec, verification, and the failure modes.
Mental model: the node stack that exposes a GPU
A GPU appears to Kubernetes only after several layers on the node are healthy.
- Driver: the vendor driver must be installed on the host, since containers use the host kernel module.
- Container runtime: NVIDIA Container Toolkit configures containerd or CRI-O so GPU devices and libraries are injected into containers.
- Device plugin: a DaemonSet registers the extended resource with the kubelet, which then reports
nvidia.com/gpuin nodecapacityandallocatable. AMD and Intel have their own plugins and resource names. - Easiest path: the NVIDIA GPU Operator automates driver, toolkit, device plugin and monitoring, while managed clouds often provide GPU node images.
- Runtime class: some setups require
runtimeClassName: nvidiain the pod spec, depending on how the toolkit and containerd were configured. The GPU Operator documents which approach it sets up by default.
Hands-on: configure GPU resources in the pod spec
Request GPUs as an extended resource under limits and keep other requests realistic.
apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
restartPolicy: Never
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base-ubuntu22.04
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1 # extended resource, integers only
- Limits are the request: extended resources are specified in
limits. If you also set a request, it must equal the limit. - Whole numbers only: you cannot request a fraction like 0.5, and GPUs are not overcommitted. Sharing uses time-slicing or MIG configured on the node.
- Tolerations and selectors: GPU nodes are usually tainted so ordinary pods stay off them, so add a matching toleration and, if needed, a nodeSelector label.
- Image: use a CUDA-compatible image whose CUDA version is supported by the host driver.
- Scheduling hints: add node labels from GPU feature discovery, for example the GPU product, so a workload can select an A100 class instead of any GPU. Label names depend on the labeller you deploy.
Verify the GPU is advertised and usable
Confirm at three levels: node capacity, scheduling and inside the container.
kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
kubectl apply -f gpu-test.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/gpu-test --timeout=120s
kubectl logs gpu-test
# Expect the nvidia-smi table listing the GPU model and driver version
- Node capacity:
kubectl describe nodeshould listnvidia.com/gpuunder Capacity and Allocatable. - Scheduling: a Pending pod with
Insufficient nvidia.com/gpuin its events means no node has free GPUs or the toleration is missing. - Container: run
nvidia-smiin the pod and check that the expected device and driver version appear. - Plugin health: check the device plugin pods and their logs if capacity shows zero.
- Check allocation:
kubectl describe nodelists allocated extended resources under Allocated resources, which shows how many GPUs are claimed and by which pods.
Failure modes and sharing options
Most GPU incidents are node-stack or scheduling problems rather than pod spec errors.
- Capacity zero: driver not loaded, toolkit not configured for the runtime, or the plugin DaemonSet not scheduled on the node (taints, selectors).
- Driver mismatch: CUDA image newer than the host driver supports yields runtime errors, so align versions.
- Exclusive allocation: one pod per GPU by default, so idle GPUs held by finished or stuck pods waste money. Use Jobs with TTL for batch work.
- Sharing: NVIDIA MIG partitions supported GPUs into isolated slices, while time-slicing shares a GPU with no memory isolation. Both are configured on the node, not in the pod.
- Multi-GPU jobs: request several devices in one container, for example
nvidia.com/gpu: 4, and remember the scheduler needs all four on one node. Topology and interconnect awareness depend on additional tooling.
Production practice: cost and isolation
GPU nodes are the most expensive part of many clusters, so control who lands on them.
- Taint GPU nodes: use a taint such as
nvidia.com/gpu=present:NoScheduleand only tolerate it in GPU workloads, often enforced with a policy. - Quota: apply
requests.nvidia.com/gpuResourceQuota per namespace to cap consumption. - Autoscale: scale GPU node groups from zero with the cluster autoscaler or Karpenter, and use priority classes for important jobs.
- Observe: export GPU metrics (for example NVIDIA DCGM exporter) and alert on utilisation, temperature and ECC errors.
- Idle detection: compare allocated GPUs with real utilisation from DCGM metrics. Pods that hold a GPU while sitting idle are a common source of waste, so review them regularly and consider automatic cleanup of finished notebooks and jobs.
- GPUs are extended resources: the host needs the driver and container toolkit, and a device plugin advertises nvidia.com/gpu to the kubelet.
- In the pod I set nvidia.com/gpu in limits as an integer, add a toleration for the GPU taint, and verify with node allocatable and nvidia-smi.
- I automate the stack with the GPU Operator, use quotas, taints and autoscaling for cost, and monitor with DCGM metrics.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Requesting a fractional GPU like 0.5 Extended resources must be integers; sharing needs MIG or time-slicing on the node.
- Putting the GPU only under requests Extended resources must be set in limits, and any request must equal the limit.
- Forgetting tolerations on tainted GPU nodes The pod stays Pending even though GPUs are free.
- Using a CUDA image newer than the host driver The container starts but CUDA calls fail at runtime.