Q: You have an architecture running 50 identical EC2 instances. You need to deploy a rapid emergency patch. Using Ansible, what is the problem with running the playbook against the inventory sequentially, and how do you speed it up?
By default, Ansible processes tasks with a concurrency of 5 forks. When patching 50 instances, sequential execution causes severe delay (20+ minutes). Overriding with -f 50 spawns 50 parallel worker processes, slashing turnaround down to ~2 minutes.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In production operations, maintenance windows during SEV-1 outages are measured in minutes. The interviewer is testing understanding of Ansible's execution model, process architecture, and concurrency limits. I structure my response around immediate triage, parallel fork optimization, and Day-2 configuration guardrails."
Analyze Default Fork Bottlenecks
By default, Ansible runs with forks = 5 in ansible.cfg. When executing against 50 hosts, Ansible divides the fleet into 10 sequential batches (50 / 5 = 10). If an emergency patching task takes 2 minutes per host, the entire deployment requires 20 minutes of critical lead time.
- Default Concurrency: 5 worker sub-processes handling SSH connections.
- Sequential Grouping: Each task waits for all 5 current hosts to complete before dispatching to the next 5 hosts.
- Blast Radius / Time Risk: Extended exposure window where instances run mixed versions.
Immediate Remediation: Override Forks at Runtime
To achieve instantaneous fleet-wide patching, override the worker forks count directly from the CLI to match the total instance count:
ansible-playbook -i inventory/aws_ec2.yml patch.yml -f 50- This instructs the Ansible controller to spawn 50 parallel Python worker processes simultaneously.
- Execution duration drops from 20 minutes down to ~2 minutes (single execution cycle).
Controller Capacity & SSH Multiplexing Tuning
Spawning 50 parallel SSH sessions requires tuning the Ansible controller OS limits and SSH configuration:
- File Descriptor Limits: Ensure
ulimit -n 65535on the controller to avoid socket exhaustion. - SSH Pipelining: Enable
pipelining = Trueinansible.cfgto execute Python modules directly via stdin without file copies. - ControlMaster: Configure
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60sto reuse persistent TCP connections.
- Default Ansible concurrency is capped at 5 forks, forcing 50 instances into 10 sequential 2-minute batches (20 min total).
- Immediately override concurrency on CLI using: ansible-playbook -i inventory patch.yml -f 50 to execute all nodes in parallel.
- Verify Ansible controller has adequate file descriptors (ulimit -n 65535) and memory headroom for 50 worker processes.
- Enable SSH pipelining = True in ansible.cfg to eliminate intermediate Python script SFTP writes.
- Use SSH ControlMaster multiplexing to persist open TCP sockets across consecutive tasks.
- Establish permanent guardrails in ansible.cfg setting default forks = 25 or 50 for CI/CD runners.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Running playbooks with default 5 forks during active P1 outages Extends incident duration by 10x. Interviewers expect senior engineers to immediately identify the concurrency parameter as the primary bottleneck.
- Increasing forks to 100+ without verifying controller CPU, memory, and ulimit Spawning too many forks on an undersized controller (e.g. t3.micro) causes fork bomb kernel OOM kills and dropped SSH handshakes.
- Confusing forks with the serial keyword Forks controls task parallelism across hosts; serial controls rolling batch deployment sizes across entire plays.