Q: Your engineering org consolidated 40 microservices into a single monorepo. Every pull request triggers all 40 test suites and Docker builds, consuming 200 runner minutes per PR and causing severe queue congestion. A PR touching only the 'auth-service' README runs the full billing deployment pipeline. How do you design an automated, accurate path-filtering and change-detection pipeline in GitHub Actions?
Engineering an intelligent change-detection and path-filtering pipeline for multi-service monorepos, running builds, tests, and deployments strictly for modified microservices.
#CI/CD #Monorepo #GitHub Actions #Path Filtering #Optimization #Git
🎙️ Candidate Opening & Architectural Context
"Monorepos without intelligent change detection waste massive compute budgets and destroy developer velocity by building unaffected services. We designed an automated path-filtering architecture using dorny/paths-filter and Git commit tree diff analysis."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Configure Declarative Path Filtering via dorny/paths-filter
Inspect changed files in the pull request against service directory patterns:
- Filter Step: Added
uses: dorny/paths-filter@v3step in a lightweight pre-flight job. - Filter Rules: Defined path globs:
auth: ['services/auth/**', 'libs/common/**'],billing: ['services/billing/**', 'libs/common/**']. - Output Matrix: Emits JSON boolean outputs:
auth: true,billing: false, indicating exactly which services have modifications.
Pro Tip: Including shared libraries (libs/common/**) in service filters ensures that updating common code triggers builds across all dependent services.
2️⃣
Generate Dynamic Job Execution Matrix Based on Filter Outputs
Trigger downstream test and build jobs strictly for modified services:
- Conditional Execution: Downstream jobs evaluate filter conditions:
if: needs.changes.outputs.auth == 'true'. - Dynamic Matrix JSON: Constructed a dynamic JSON array of changed services passed into a reusable workflow matrix:
services: ${{ fromJSON(needs.changes.outputs.modified_services) }}.
Pro Tip: Using dynamic matrix generation allows a single parameterized build workflow to run in parallel only for modified services.
Advertisement
3️⃣
Handle Git Merge-Base & Squash Merge Edge Cases
Ensure accurate change detection across rebased and squashed branches:
- Merge-Base Diffing: Configured path diffing against the common merge-base ancestor:
git diff --name-only $(git merge-base origin/main HEAD) HEAD. - Force Rebuild Flag: Added support for PR label
ci:force-all; if present, the filter overrides path detection and triggers builds across all 40 services.
Pro Tip: Calculating diffs against git merge-base ensures that long-lived feature branches don't miss changes committed on main.
4️⃣
Measure CI Runner Queue Reductions & Feedback Speed
Audit pipeline execution speed and compute bill reduction:
- Duration Reduction: Average PR check runtime plummeted from 28 minutes down to 3 minutes 20 seconds.
- Compute Savings: Total GitHub Actions runner minute consumption dropped by 74%, slashing monthly cloud CI invoices by over $16,000.
- Developer Satisfaction: Engineers receive PR green checkmarks in minutes, unblocking deployment release trains.
Pro Tip: Smart path filtering unlocks the collaboration benefits of a monorepo without the painful CI pipeline performance penalty.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Monorepo CI acceleration uses Git merge-base diffing and path-filtering actions to build only modified services and shared libraries, cutting runner consumption by over 70%."
⚡ 60-Second Elevator Pitch Talking Points
- Inspect PR commit diffs against shared libraries and service directories using paths-filter.
- Generate dynamic JSON matrix outputs to run CI jobs only for modified services.
- Diff against git merge-base to prevent missing changes on rebased feature branches.
- Provide a ci:force-all PR label escape hatch while slashing CI runner minutes by 74%.
Advertisement