Q: Your Lambda function is doing the same cold start every invocation because it initializes a big ML model. How do you fix it?
To fix a Lambda cold start with a big ML model, load the model once in the init phase (outside the handler), shrink and cache it, and pay to keep environments warm with provisioned concurrency where latency matters.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: To fix a <strong>Lambda cold start with a big ML model</strong>, load the model once in the init phase (outside the handler), shrink and cache it, and pay to keep environments warm with provisioned concurrency where latency matters. If the function seems to cold start on every invocation, either the model is loaded inside the handler, or concurrency is so spiky that each request gets a new execution environment. Interviewers ask this to test whether you understand the Lambda lifecycle (init, invoke, freeze) and the real options: global scope reuse, packaging choices, provisioned concurrency, SnapStart where the runtime supports it, and when Lambda is the wrong tool.
Understand the lifecycle: what repeats and what does not
Only code outside the handler runs once per execution environment; the handler runs per invocation.
- Init phase: downloads code, starts the runtime and runs top-level code. A cold start is this phase plus the first invoke. It appears as
Init Durationin the REPORT log line only on cold starts. - Warm invokes: reuse the same environment, its global variables and
/tmp, until it is idle long enough to be reclaimed. - Diagnosis question: if every REPORT line shows Init Duration, either each request lands on a new environment (bursty concurrency or low traffic) or you are not caching anything.
- Memory is CPU: more memory gives proportionally more CPU, which often shortens model deserialization considerably.
- Why it looks like every call: at low or very spiky traffic, Lambda scales out new environments for concurrent requests, and idle ones are eventually reclaimed, so sparse traffic means frequent cold starts.
Hands-on: measure the Lambda cold start with Logs Insights
Quantify how often cold starts happen and how long init takes before optimizing.
# CloudWatch Logs Insights query (run against /aws/lambda/my-ml-fn)
fields @timestamp, @initDuration, @duration, @maxMemoryUsed
| filter @type = "REPORT"
| stats count() as invokes, count(@initDuration) as cold_starts,
avg(@initDuration) as avg_init_ms, pct(@initDuration, 95) as p95_init_ms
by bin(1h)
- Cold start ratio: count REPORT lines that include an init duration versus the total.
- Where time goes: add timing logs around model download, deserialization and first inference.
- X-Ray or Lambda Insights: enable active tracing to see the initialization subsegment.
- Set a target: decide an acceptable p95 for first-request latency before choosing between code fixes and paid warm capacity.
Fix 1: load the model once, in init scope
Move heavy initialization out of the handler and make it lazy-safe.
import os, boto3, onnxruntime as ort
MODEL_PATH = "/tmp/model.onnx"
_session = None
def _load():
global _session
if _session is None:
if not os.path.exists(MODEL_PATH):
boto3.client("s3").download_file("my-models", "v3/model.onnx", MODEL_PATH)
_session = ort.InferenceSession(MODEL_PATH)
return _session
_load() # run in init phase so provisioned concurrency pre-warms it
def handler(event, context):
return {"score": float(_load().run(None, {"x": event["features"]})[0][0])}
- Global variable: load at module import so later invocations reuse it.
- Local artifact: bake the model into the container image, or fetch from S3 to
/tmponce and check for the file before downloading again (ephemeral storage is configurable up to 10 GB). - Smaller model: quantize, prune or export to ONNX; load only what inference needs. Package size affects init time.
- Container image: images up to 10 GB are supported and avoid zip size limits, but a large image still needs to be loaded on first use.
- Thread-safety: the loaded object is shared across invocations in one environment, but each environment handles one request at a time, so ordinary inference code is safe.
Fix 2: remove the cold start from the user path
When init cannot be shortened enough, pre-initialize environments.
aws lambda publish-version --function-name my-ml-fn
aws lambda create-alias --function-name my-ml-fn --name live --function-version 7
aws lambda put-provisioned-concurrency-config --function-name my-ml-fn \
--qualifier live --provisioned-concurrent-executions 5
aws lambda get-provisioned-concurrency-config --function-name my-ml-fn --qualifier live
# Status should become READY; point callers at the alias ARN
- Provisioned concurrency: keeps a set number of environments initialized, including your top-level model load. It must target a version or alias, not
$LATEST, and is billed while configured; use Application Auto Scaling to schedule it. - SnapStart: snapshots the initialized environment and resumes from it. Support depends on runtime (Java originally, with Python and .NET added later) and region, so verify current support before relying on it.
- Not a fix: reserved concurrency only caps or guarantees capacity; it does not keep anything warm. Periodic ping hacks are fragile compared with provisioned concurrency.
- Right tool: for multi-GB models or GPU needs, a SageMaker endpoint or container service is usually better than Lambda.
- Autoscale warm capacity: register the alias with Application Auto Scaling using target tracking on utilization, or scheduled actions for known peaks.
Verify and operate
Prove the improvement and keep it.
- Re-run the Insights query after deployment; cold_starts should fall and p95 latency should drop. Metric
ProvisionedConcurrencySpilloverInvocationsshows traffic exceeding the pre-warmed pool. - Init limits: the init phase has its own time limit, so a model that loads too slowly will fail outright rather than just be slow. Check current limits in the docs.
- Version the model: deploy new model versions through new function versions so aliases shift traffic gradually.
- Cost check: compare provisioned concurrency spend against the latency requirement; scheduled scaling covers business hours only.
- Layers and dependencies: trim heavy libraries (full TensorFlow or PyTorch) in favor of lighter runtimes such as ONNX Runtime to cut package size.
- Cold start cost is the init phase, so a big ML model must load once at module scope, not inside the handler.
- I measure with Logs Insights on Init Duration, shrink the model, cache it in the image or /tmp, then use provisioned concurrency on an alias.
- I automate scheduled provisioned concurrency scaling and alarm on spillover invocations and p95 latency.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Loading the model inside the handler Every invocation pays the full load cost, even on warm environments.
- Applying provisioned concurrency to $LATEST It only works on a published version or alias.
- Assuming reserved concurrency keeps functions warm It limits and guarantees capacity but does not pre-initialize environments.
- Forcing a multi-GB model into Lambda Init time limits and memory ceilings make SageMaker or a container service the better fit.