Q: An EC2 instance is running normally, and the application is also working as expected. However, the application team has raised a Sev-1 incident because they are unable to SSH into the EC2 instance. You have joined the Sev-1 call. What areas would you check to restore SSH connectivity?
Sev-1 incident response procedure when an EC2 instance application operates normally, but operations engineers cannot establish SSH sessions.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Because the application is functioning normally, the underlying EC2 hardware, hypervisor, OS kernel, and primary application ports are healthy. The failure is isolated strictly to the SSH management path. I troubleshoot in order from external network perimeter to host OS configuration.
Check Security Group & Network ACL (NACL) Ingress Rules
Verify port 22 ingress rules on the EC2 instance's Security Group. Check if someone modified the corporate VPN CIDR or accidentally deleted the port 22 rule. Verify that the subnet NACL allows inbound port 22 and outbound ephemeral ports (1024-65535).
# Verify Security Group rules via AWS CLI
aws ec2 describe-security-groups \
--group-ids sg-0123456789abcdef0 \
--query "SecurityGroups[*].IpPermissions[?ToPort==\`22\`]"
Verify Routing Table & Bastion / VPN Path
If engineers connect via a Bastion host or AWS Client VPN, verify that the Bastion instance is operational and its route table directs traffic through the Internet Gateway or Transit Gateway.
aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=subnet-01234567"
Bypass SSH via AWS Systems Manager (SSM) Session Manager
Do NOT waste precious Sev-1 time debugging port 22 if you need immediate terminal access. Connect directly using AWS SSM Session Manager. SSM relies on the outbound SSM agent and does not require port 22, public IPs, or security group ingress rules.
# Instantly connect via SSM bypassing SSH entirely
aws ssm start-session --target i-0123456789abcdef0
Inspect Host sshd Daemon, iptables, and Disk Space
Once inside via SSM or EC2 Serial Console:
1. Check systemctl status sshd (is sshd running or listening on an alternate port?).
2. Check journalctl -u sshd -n 50 for authentication failures or MaxStartups drops.
3. Check df -h (if root / is 100% full, sshd cannot allocate pty sessions and immediately terminates connections).
- Check Security Group and Subnet NACL ingress rules for port 22 and corporate VPN CIDRs.
- Use AWS SSM Session Manager to immediately bypass SSH and access the terminal within seconds.
- Use the EC2 Serial Console if the network stack is completely unresponsive.
- Inspect sshd daemon status, root disk space (/tmp and /), and host iptables/firewalld rules.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Purging compromised IAM user credentials immediately prior to querying CloudTrail forensic event logs when configuring Connectivity Incident Application Deactivating a compromised key prevents abuse while preserving identity records for CloudTrail forensic queries to assess blast radius.
- Conflating local IAM Permissions Boundaries with organizational Service Control Policies (SCPs) when configuring Connectivity Incident Application SCPs define maximum permission guards at the AWS Organization level across accounts, whereas permission boundaries limit delegate privileges within a single account.
- Deploying single-AZ cloud infrastructure for production workloads without multi-AZ failover resilience when configuring Connectivity Incident Application Availability Zone hardware failures or localized network partitions cause complete service downtime when resources are not distributed across >=2 AZs.
- Jumping straight to SSH or instance restarts without reviewing CloudWatch metrics and AWS Systems Manager logs when configuring Connectivity Incident Application Senior SREs triage control plane and hypervisor metrics via CloudWatch and SSM Session Manager before executing invasive host-level actions.