⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 114 of 187 in CI/CD & GitOps
Senior DevOps / SRE CI/CD GitHub Actions Production Incident
🎯 Target Role / Context: Senior DevOps & CI/CD Pipeline Reliability Triage

Q: How do you troubleshoot, triage, and handle failed workflows in GitHub Actions?

Systematic workflow failure triage: categorizing failures into code, transient infrastructure, or pipeline design defects, using the GitHub CLI (gh) for rapid inspection, re-running failed jobs, and implementing automated retries.

#CI/CD #GitHub Actions #Troubleshooting #gh CLI #Debugging #Retries #Resilience
🎙️ Candidate Opening & Architectural Context
"I treat workflow failures as either code defects, transient infrastructure/external dependency issues, or pipeline design bugs. First I inspect the failed job logs and artifacts using the gh CLI, rerun failed jobs for transient network issues, and correlate with external provider outages. Then I fix the root cause and implement guardrails like step retries, cache optimizations, debug logging, or quarantining flaky tests."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: I treat workflow failures as either code defects, transient infrastructure/external dependency issues, or pipeline design bugs. First I inspect the failed job logs and artifacts using the gh CLI, rerun failed jobs for transient network issues, and correlate with external provider outages. Then I fix the root cause and implement guardrails like step retries, cache optimizations, debug logging, or quarantining flaky tests.

1️⃣

Rapid Triage Using the GitHub CLI (gh)

Diagnose workflow failures quickly without clicking through multiple web UI menus:

# List recent pipeline runs
gh run list --limit 20

# View only the failed log output directly in terminal
gh run view 12345678 --log-failed

# Download diagnostic artifacts
gh run download 12345678 --dir ./artifacts/

# Re-run only the failed jobs in the workflow
gh run rerun 12345678 --failed
  • Inspect Failed Logs: Use gh run view <run-id> --log-failed to output only the exact failing step lines.
  • Download Artifacts: Retrieve test reports, screenshots, and logs using gh run download.
  • Rerun Only Failed Jobs: Avoid re-executing successful 30-minute test jobs by using gh run rerun --failed to retry only the affected component.
  • Debug Logging: Enable runner and step diagnostic logging by setting repository secrets ACTIONS_STEP_DEBUG=true and ACTIONS_RUNNER_DEBUG=true.
2️⃣

Implementing Pipeline Resilience & Automated Retries

Prevent recurring transient failures from blocking development velocity:

# Example automated retry for network-sensitive step
- name: Push container image with exponential retry
  uses: nick-fields/retry@v3
  with:
    timeout_minutes: 5
    max_attempts: 3
    retry_wait_seconds: 10
    command: docker push ghcr.io/org/api:${{ github.sha }}
  • Step-Level Retries: Use retry actions (such as nick-fields/retry) for network-sensitive operations like Docker image pushes or registry downloads.
  • API Rate Limit Mitigation: Authenticate all external API calls (e.g. GitHub API, Docker Hub) using personal or bot tokens to avoid anonymous IP rate limits.
  • Flaky Test Isolation: Quarantine flaky end-to-end tests into a non-blocking job or rerun failed tests once before failing the entire pipeline.
  • Notification Webhooks: Configure Slack / PagerDuty alert webhooks triggered only on if: failure() to notify on-call engineers immediately.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Categorize failures into code vs transient vs configuration bugs. Use the gh CLI for fast log extraction and rerun-failed operations, and add automated step retries for network-sensitive registry and package downloads."
⚡ 60-Second Elevator Pitch Talking Points
  • Diagnose pipeline failures rapidly with gh run view --log-failed and re-run only failed jobs.
  • Enable ACTIONS_STEP_DEBUG and ACTIONS_RUNNER_DEBUG to uncover hidden runner and network issues.
  • Harden pipelines against transient failures with exponential retries and authenticated registry access.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Handling Failed GitHub Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
  2. Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Handling Failed GitHub Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
  3. Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Handling Failed GitHub Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
  4. Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Handling Failed GitHub Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →