Q: A new firmware build pipeline starts failing during artifact uploads. S3 shows no error, Jenkins logs are incomplete. Where do you start?
Triage runbook for resolving high-value firmware build pipeline upload failures where AWS S3 shows no error logs and Jenkins output is truncated.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a build pipeline producing large binaries (such as multi-gigabyte vehicle firmware images) fails silently during S3 upload without S3 errors or complete Jenkins logs, the failure occurs client-side before the HTTP request completes: Jenkins agent out-of-memory (OOM) killed by the Linux kernel, ephemeral disk space depletion, or AWS CLI multi-part upload timeout.
Check Linux Kernel dmesg for OOM-Killer Execution
When Jenkins logs truncate abruptly without a stack trace, the process was killed by SIGKILL (signal 9) rather than exiting gracefully. Check dmesg or system logs on the build node for Out of memory: Killed process.
# Check for OOM killer termination on Jenkins agent
dmesg -T | grep -i -E 'oom[-_]killer|killed process'
# If on Kubernetes runner:
kubectl describe pod <jenkins-agent-pod> | grep -i 'OOMKilled'
Inspect Agent Ephemeral Disk Space & Temp Inodes
The AWS CLI and SDKs buffer large multi-part uploads in the local temporary directory (/tmp) before splitting into 5 MB chunks. If the agent disk hits 100%, the process crashes instantly.
df -h /tmp
df -i /tmp
Enable Verbose AWS CLI Debug Telemetry
Re-run the upload command with --debug. This exposes exact TCP socket timeouts, SSL handshake failures, and IAM STS token expiration during long-running uploads.
aws s3 cp firmware.bin s3://tesla-firmware-builds/ --debug > upload_debug.log 2>&1
Tune Multi-Part Upload Concurrency and Chunk Size
Adjust AWS S3 CLI configurations to handle massive firmware binaries safely:
Increase multipart_chunksize to 64MB and tune max_concurrent_requests to avoid saturating agent memory and file descriptors.
aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 64MB
aws configure set default.s3.max_concurrent_requests 10
- Check dmesg -T on the agent for Linux OOM-killer terminating the upload process with SIGKILL.
- Verify local agent /tmp disk space and inode availability.
- Execute the upload with the aws s3 --debug flag to uncover socket resets or expired STS tokens.
- Tune AWS S3 multipart_chunksize to 64MB and control concurrency to reduce memory overhead.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Failing to configure lifecycle expiration rules for incomplete multipart S3 uploads Aborted or incomplete multipart upload parts remain in storage indefinitely, accumulating storage costs without appearing in standard console bucket listings.
-
Attempting cross-account S3 object copies without configuring destination bucket ownership overrides
By default, objects uploaded across accounts remain owned by the uploader account unless
bucket-owner-full-controlis explicitly enforced. - Relying on public internet S3 endpoints from inside private VPC subnets instead of S3 Gateway Endpoints VPC S3 Gateway Endpoints eliminate NAT Gateway data processing fees and ensure high-throughput storage traffic remains entirely on the private AWS network.
- Enabling S3 Cross-Region Replication (CRR) without verifying versioning status on both source and destination buckets in tesla-q005-firmware-build-pipeline-s3-artifact-upload-failure CRR strictly requires S3 Versioning to be enabled on both source and destination buckets to synchronize object revisions and delete markers.