Q: Which specific Prometheus metrics from NVIDIA DCGM Exporter distinguish true GPU compute utilization from memory-copy stalls, and how do you detect NVLink bandwidth degradation and thermal clock throttling across thousands of GPUs?
Deploying and configuring NVIDIA Data Center GPU Manager (DCGM) Exporter on Kubernetes, instrumenting critical Prometheus metrics for SM utilization, NVLink bandwidth, and thermal/power throttling.
DCGM_FI_DEV_GPU_UTIL metric can be deeply misleading: it reports 100% if a GPU is merely waiting on memory transfers from host RAM while compute cores sit completely idle. Engineering robust GPU observability requires capturing low-level hardware counters exposed by NVIDIA Data Center GPU Manager (DCGM)."
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Standard Kubernetes metrics (CPU and RAM) tell you nothing about GPU workloads. Furthermore, the generic DCGM_FI_DEV_GPU_UTIL metric can be deeply misleading: it reports 100% if a GPU is merely waiting on memory transfers from host RAM while compute cores sit completely idle. Engineering robust GPU observability requires capturing low-level hardware counters exposed by NVIDIA Data Center GPU Manager (DCGM).
Deploy NVIDIA DCGM Exporter DaemonSet with Custom Profiling Counters
Deploy DCGM Exporter via the NVIDIA GPU Operator or Helm chart. By default, DCGM Exporter uses a basic metrics list. Upgrade to the profiling metrics configuration (dcp-metrics-included.csv), enabling hardware performance counters for Streaming Multiprocessor (SM) active cycles, tensor core activity, and memory bus utilization.
# helm values for dcgm-exporter
serviceMonitor:
enabled: true
interval: 15s
arguments:
- "-f"
- "/etc/dcgm-exporter/dcp-metrics-included.csv"
Differentiate Compute vs Memory Stalls: SM Utilization vs Tensor Pipe Active
To measure true model compute efficiency, correlate: DCGM_FI_PROF_SM_ACTIVE (fraction of time SMs have at least one warp executing) and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (fraction of time Tensor Cores are active, indicating FP16/BF16/FP8 matrix multiplications) against DCGM_FI_PROF_DRAM_ACTIVE (memory bandwidth saturation). If SM_ACTIVE is low while DRAM_ACTIVE is 90%+, the workload is memory-bandwidth bound rather than compute bound.
# Prometheus PromQL for true Tensor Core utilization percentage
avg(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) by (pod, namespace) * 100
Detect Hardware Throttling and NVLink Bottlenecks
Instrument alerts for GPU hardware throttling: monitor DCGM_FI_DEV_CLOCK_THROTTLE_REASONS bitmasks. Bit 0x0000000000000008 indicates thermal throttling (temperatures exceeding thermal limits), and bit 0x0000000000000004 indicates power brake throttling. Track DCGM_FI_PROF_NVLINK_TX_BYTES and DCGM_FI_PROF_NVLINK_RX_BYTES to detect imbalanced tensor parallel communication across GPU ranks.
# Alert rule: GPU Thermal or Power Throttling detected
- alert: GPUClockThrottled
expr: DCGM_FI_DEV_CLOCK_THROTTLE_REASONS > 0
for: 2m
labels:
severity: warning
annotations:
summary: "GPU {{ $labels.gpu }} on {{ $labels.node }} is throttling clock frequencies"
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE to measure real Tensor Core math execution, track DRAM_ACTIVE for memory bus saturation, and alert on CLOCK_THROTTLE_REASONS to catch thermal and power degradation."
- Basic
nvidia-smiutilization is misleading because waiting on memory transfers shows up as 100% utilized. - We configure DCGM Exporter with profiling metrics to track
SM_ACTIVEandPIPE_TENSOR_ACTIVE, revealing exactly how much time GPUs spend crunching matrix math. - We alert on
CLOCK_THROTTLE_REASONSand monitor NVLink TX/RX byte throughput to catch thermal throttling and interconnect bottlenecks before they ruin training schedules.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Deploying multi-GPU tensor parallelism across standard PCIe interconnects rather than high-bandwidth NVLink bridges Tensor Parallelism across PCIe switches instead of NVLink bridges severely degrades token throughput due to bus bandwidth saturation.
- Neglecting host NVIDIA kernel driver compatibility matrices when updating Kubernetes GPU Operator images Driver and CUDA runtime version mismatches prevent the NVIDIA GPU Operator from initializing GPU worker nodes properly.
- Overlooking thermal and electrical throttling metrics exported by the NVIDIA DCGM telemetry agent Thermal and power throttling silently drop GPU clock frequencies; DCGM metrics expose hidden hardware degradation.
- Routing inference requests to newly scheduled model containers before initial GPU memory warmup completes Routing user inference requests to pods before model weights are loaded into VRAM causes immediate client timeouts and 503 errors.