Q: A FastAPI/PyTorch inference container starts at 12GB VRAM usage, but steadily climbs by 50MB per request until triggering a fatal CUDA OOM after 12 hours. How do you systematically trace and fix this memory leak?
Isolating and resolving progressive VRAM memory leaks in long-running Python/PyTorch inference microservices caused by autograd graphs, dangling tensor references, and uncollected CUDA caches.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: In Python and PyTorch, GPU memory leaks are rarely caused by native CUDA driver bugs; they almost always stem from Python object retention. When an application inadvertently preserves references to output tensors, the entire computation history or PyTorch autograd graph remains pinned in GPU memory, preventing the PyTorch caching allocator and Python garbage collector from reclaiming the blocks.
Enforce Inference Contexts: torch.no_grad() and model.eval()
Ensure the model is switched to evaluation mode (model.eval()) and every forward execution is wrapped in with torch.no_grad(): or with torch.inference_mode():. Without inference_mode, PyTorch builds intermediate gradient computation graphs in VRAM for backpropagation, bloating memory with every request.
# Correct inference execution pattern
model.eval()
with torch.inference_mode():
outputs = model(inputs)
Prevent Python Reference Leaks: Detach Tensors and Extract Primitives
Inspect logging and metrics code. Storing output tensors in global request logs (e.g., history.append(loss) or metrics.append(pred)) keeps the tensor and its underlying GPU memory alive. Always call .detach().cpu().item() to convert GPU tensors into native Python numbers before appending to telemetry data structures.
# Leak:
# metric_history.append(output_tensor)
# Fix:
metric_history.append(output_tensor.detach().cpu().item())
Profile VRAM and Object Allocations with PyTorch Memory Profiler and Tracemalloc
Instrument the service with torch.cuda.memory._record_memory_history() and export snapshot dumps viewable at pytorch.org/memory_viz. Correlate with Python's tracemalloc to identify the exact line of code holding lingering tensor references across HTTP request boundaries.
import torch
torch.cuda.memory._record_memory_history(max_entries=100000)
# Dump trace after 500 requests
torch.cuda.memory._dump_snapshot("/tmp/memory_snapshot.pickle")
torch.inference_mode() or saving raw GPU tensors to in-memory logs. Use .detach().cpu().item() for metrics and analyze memory snapshots via pytorch.org/memory_viz."
- GPU memory climbing steadily in inference is usually a Python object lifecycle bug, not a CUDA driver issue.
- Running without
torch.inference_mode()forces PyTorch to maintain full autograd graphs for every request. - Saving tensors into metric arrays pins entire GPU memory buffers; converting values with
.detach().cpu().item()and profiling via PyTorch memory snapshots completely eliminated our creep.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing GPU streaming multiprocessor utilization with raw VRAM video memory allocation when configuring Debugging Memory Leaks A GPU can show 100% VRAM memory allocation while compute cores remain completely idle due to CPU preprocessing bottlenecks or batch starvation.
- Running LLM inference endpoints without PagedAttention KV-cache fragmentation controls in vLLM when configuring Debugging Memory Leaks Unmanaged KV cache allocation results in frequent CUDA Out-of-Memory (OOM) errors and limits concurrent request batching capacity.
- Employing static batching on inference models instead of continuous dynamic request batching (Triton) when configuring Debugging Memory Leaks Static batching forces requests to wait until batch windows fill, introducing massive latency spikes on variable real-time traffic.
- Baking massive multi-gigabyte neural network weights directly inside application container images when configuring Debugging Memory Leaks Baking 50GB weights into image layers creates enormous pod startup times, exhausts node disk space, and blocks rapid scaling.