Q: Your engineering org has 200 developers running intensive build jobs that require access to private databases, internal SonarQube, and Docker daemon builds. Microsoft-hosted agents are too slow and cannot access your private VNet. How do you design and deploy auto-scaling, ephemeral VMSS self-hosted agent pools?
Engineering secure, auto-scaling ephemeral self-hosted CI/CD build agents in private Azure VNets using Azure DevOps Scale Set agent pools with automated custom VM image baking.
#Azure #Azure DevOps #VMSS #CI/CD #Self-Hosted Agents #VNet
🎙️ Candidate Opening & Architectural Context
"Static self-hosted VM build agents accumulated leftover build artifacts, suffered from configuration drift, and ran 24/7 wasting compute budget. We re-engineered our CI/CD platform using Azure DevOps VMSS self-hosted agent pools."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Bake Hardened Golden VM Image using Packer & Azure Compute Gallery
Pre-install heavyweight build dependencies (Docker, Java, .NET, Node, Helm) to slash pipeline setup times:
- Packer Template: Automated Ubuntu 22.04 VM image baking with Docker engine, build runtimes, and security hardening.
- Compute Gallery: Published image to Azure Compute Gallery with multi-region replication:
az sig image-version create --gallery-name ci_gallery --image-definition agent-image --version 1.2.0.
Pro Tip: Pre-baking runtimes into the VM image cuts pipeline execution time by up to 6 minutes per build run.
2️⃣
Provision Azure VMSS in Private Subnet
Deploy an autoscaling Virtual Machine Scale Set with zero public IP addresses:
- VMSS Provisioning: Created VMSS using
az vmss create --name vmss-agents --resource-group rg-ci --image $IMAGE_ID --vm-sku Standard_D4s_v5 --subnet $SUBNET_ID --public-ip-address '' --upgrade-policy-mode Manual --instance-count 0. - Managed Identity: Assigned User-Assigned Managed Identity with ACR Pull and Key Vault Read permissions.
Pro Tip: Setting --public-ip-address '' ensures build agents reside purely inside RFC 1918 private subnets with internet egress managed via NAT Gateway.
3️⃣
Register VMSS Agent Pool in Azure DevOps
Connect Azure DevOps project settings directly to the Azure VMSS resource:
- Create Pool: Configured new Agent Pool in Azure DevOps Project Settings with Pool Type = 'Azure virtual machine scale set'.
- Tuning Parameters: Configured Maximum machines = 30, Standby instances = 2, and enabled 'Automatically tear down virtual machines after every use'.
Pro Tip: Tearing down VMs after every job guarantees a pristine, ephemeral build environment free of cached credentials or leftover files.
4️⃣
Run Pipelines & Monitor Agent Elasticity and Cost
Validate pipeline dispatch and autoscaling scale-up and scale-down triggers:
- Pipeline YAML: Targeted pool with
pool: { name: 'VMSS-SelfHosted-Pool' }. - Auto-Scaling Metrics: Confirmed that Azure DevOps spawns new VM instances during peak morning pull-request hours and scales down to standby instances at night.
Pro Tip: Auto-scaling VMSS agents delivered a 55% monthly cost reduction compared to 24/7 static self-hosted VMs.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Azure DevOps VMSS agent pools deliver the security of private VNet access, the purity of ephemeral single-use VMs, and the cost efficiency of automated elastic scaling."
⚡ 60-Second Elevator Pitch Talking Points
- Bake hardened build toolchains into golden VM images using Azure Compute Gallery.
- Deploy a private Virtual Machine Scale Set with zero public IPs inside an enterprise VNet.
- Register the VMSS as an Azure DevOps agent pool with ephemeral VM teardown after every job.
- Cut compute costs by 55% while guaranteeing zero credential leakage between pipeline runs.
Advertisement