Q: Jenkins jobs are randomly failing at the artifact upload step. What layers would you check?
Triage runbook for diagnosing intermittent failures during artifact uploads (Nexus, Artifactory, S3) from Jenkins pipeline agents.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Layer 1: Inspect Jenkins Agent Local Ephemeral Disk & Inodes
Check whether the ephemeral agent node ran out of disk space or inodes during the build. Upload tools often stage tarball archives in `/tmp` before transmitting; if the agent disk hits 100%, the process throws an unhandled I/O exception.
df -h /tmp
df -i /tmpLayer 2: Network Firewalls, Idle Timeouts & MTU Fragmentation
Examine firewall and load balancer logs between Jenkins agents and JFrog Artifactory / Sonatype Nexus. If uploading a 2 GB file over a slow network takes 10 minutes, intermediate stateful firewalls or reverse proxies with an idle timeout of 300s will drop the TCP session midway with 'Connection reset by peer'.
# Test sustained bandwidth and latency from agent to repository
curl -w "@curl-format.txt" -T large-test-file.bin -u user:pass https://nexus.internal/repo/Layer 3: Artifact Repository Server Thread Pool & Connection Limits
Inspect the Artifactory/Nexus server metrics. If multiple concurrent Jenkins jobs complete simultaneously and attempt parallel uploads, the repository's HTTP server (NGINX/Tomcat) exhausts its max worker threads, returning HTTP 503 or 429.
Layer 4: Storage Backend Quotas & IAM Token Expiry
Check if the underlying artifact storage backend (S3 bucket or Azure Blob) is hitting rate limits or if the temporary cloud IAM role credential (AWS STS / Azure OIDC token) expired during a long-running build right before the upload step began.
- Check reverse proxy (NGINX/ALB) client_max_body_size and idle connection timeout settings.
- Verify Jenkins ephemeral agent /tmp disk space and inode availability.
- Inspect Artifactory/Nexus thread pool saturation and storage backend quotas under peak build traffic.
- Configure upload clients with automated retry mechanisms and exponential backoff.