🎯 Target Role / Context:
Junior / Associate DevOps Scenario #9 · Network & SSH Triage
Q: During a scheduled maintenance window, ansible all -m ping reports 'UNREACHABLE' for 30 out of 100 Linux servers. Walk me through your step-by-step diagnostic triage runbook using SSH verbosity, bastions, and ControlMaster.
Triage UNREACHABLE errors by isolating network connectivity, SSH authentication, bastion proxy hops, and Python interpreter paths. Use -vvvv to inspect raw SSH commands and ControlMaster sockets.
#Ansible #SSH #Troubleshooting #Networking #UNREACHABLE
🎙️ Candidate Opening & Architectural Context
""In fleet management, 'UNREACHABLE' is the most common operational error. It typically stems from network security groups, expired SSH certificates, bastion proxy jumps, or missing remote Python interpreters. I approach this using systematic bottom-up network and SSH debugging.""
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Run Maximum Verbosity to Inspect Raw SSH
Execute the ping module against a single unreachable host with full -vvvv debug output:
ansible failed_host -m ping -vvvv- Inspect the exact OpenSSH command generated by Ansible.
- Identify failure stage: TCP connection timeout, host key verification mismatch, or permission denied (publickey).
2️⃣
Validate Layer 3/4 Network & Security Groups
Confirm basic network reachability from the controller to the target node:
- Check port 22 reachability:
nc -zv -w 322 - If timeout occurs: verify AWS VPC Security Groups, Route Tables, and NACLs.
- Verify bastion ProxyJump configuration in
ansible.cfg:ProxyJump=bastion.corp.internal.
Advertisement
3️⃣
Verify Authentication and Python Interpreter
If SSH connects but Ansible still fails, inspect permissions and runtime binaries:
- Test direct SSH login with Ansible key:
ssh -i ~/.ssh/id_rsa user@target-ip - Verify remote Python path: Ansible requires Python 3 on target hosts. If located in non-standard paths, configure
ansible_python_interpreter: /usr/bin/python3in host variables.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"An UNREACHABLE error points to network routing, SSH keys, or Python binary availability. Use -vvvv to isolate the exact SSH command, verify port 22 reachability, and check ansible_python_interpreter path definitions."
⚡ 60-Second Elevator Pitch Talking Points
- Isolate an unreachable host and run 'ansible host -m ping -vvvv' to view the raw OpenSSH handshake.
- Check Layer 4 network connectivity using 'nc -zv <target-ip> 22' to rule out AWS Security Groups and NACLs.
- Verify SSH key authentication and permissions (chmod 600 id_rsa) on the controller.
- Check Bastion/Jump host ProxyJump settings in ssh_args within ansible.cfg.
- Confirm that Python 3 is installed on the remote host and declare 'ansible_python_interpreter: /usr/bin/python3'.
- Clear stale SSH ControlMaster sockets in ~/.ansible/cp/ that can cause persistent connection hangs.
⛔ Critical Candidate Mistakes to Avoid in This Scenario
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Testing ping against all 30 hosts at once instead of isolating one host with -vvvv Generates unreadable console noise; senior engineers isolate a single representative failure to identify root cause.
- Assuming ping module uses ICMP packets Ansible's ping module executes an SSH login and runs a small Python snippet; it requires SSH access and Python on targets, not ICMP.
- Ignoring stale ControlMaster multiplexing sockets Old sockets in ~/.ansible/cp/ cause persistent connection failures even after network issues are resolved.
Advertisement