Q: A FastAPI/PyTorch inference container starts at 12GB VRAM usage, but steadily climbs by 50MB per request until triggering a fatal CUDA OOM after 12 hours. How do you systematically trace and fix this memory leak?
Isolating and resolving progressive VRAM memory leaks in long-running Python/PyTorch inference microservices caused by autograd graphs, dangling tensor references, and uncollected CUDA caches.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Enforce Inference Contexts: torch.no_grad() and model.eval()
Ensure the model is switched to evaluation mode (`model.eval()`) and every forward execution is wrapped in `with torch.no_grad():` or `with torch.inference_mode():`. Without `inference_mode`, PyTorch builds intermediate gradient computation graphs in VRAM for backpropagation, bloating memory with every request.
# Correct inference execution pattern
model.eval()
with torch.inference_mode():
outputs = model(inputs)Prevent Python Reference Leaks: Detach Tensors and Extract Primitives
Inspect logging and metrics code. Storing output tensors in global request logs (e.g., `history.append(loss)` or `metrics.append(pred)`) keeps the tensor and its underlying GPU memory alive. Always call `.detach().cpu().item()` to convert GPU tensors into native Python numbers before appending to telemetry data structures.
# Leak:
# metric_history.append(output_tensor)
# Fix:
metric_history.append(output_tensor.detach().cpu().item())Profile VRAM and Object Allocations with PyTorch Memory Profiler and Tracemalloc
Instrument the service with `torch.cuda.memory._record_memory_history()` and export snapshot dumps viewable at `pytorch.org/memory_viz`. Correlate with Python's `tracemalloc` to identify the exact line of code holding lingering tensor references across HTTP request boundaries.
import torch
torch.cuda.memory._record_memory_history(max_entries=100000)
# Dump trace after 500 requests
torch.cuda.memory._dump_snapshot("/tmp/memory_snapshot.pickle")- GPU memory climbing steadily in inference is usually a Python object lifecycle bug, not a CUDA driver issue.
- Running without `torch.inference_mode()` forces PyTorch to maintain full autograd graphs for every request.
- Saving tensors into metric arrays pins entire GPU memory buffers; converting values with `.detach().cpu().item()` and profiling via PyTorch memory snapshots completely eliminated our creep.