Q: SSH currently uses password authentication for access to production servers. Describe a certificate-based SSH authentication system for managing access to 1000+ servers without relying on password distribution or key rotation complexity.
Certificate-based SSH (using OpenSSH certificates) eliminates password management and simplifies key rotation across large deployments.
#Linux #Linux / SRE — Scenario-Based Interview Questions #L2 #SRE #Systems #Troubleshooting
🎙️ Candidate Opening & Architectural Context
""We encountered this OS-level bottleneck during peak traffic and diagnosed it down to kernel and filesystem metrics. The interviewer is testing: SSH security architecture, certificate-based auth, key management at scale.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
Certificate-based SSH (using OpenSSH certificates) eliminates password management and simplifies key rotation across large deployments.
- CA Setup: Designate a secure certificate authority machine that signs user and host SSH keys. Generate a CA key pair:
- User Authentication: Instead of distributing individual keys, users request a signed certificate:
- User generates their personal key:
ssh-keygen -t rsa -f ~/.ssh/id_rsa - User submits their public key to the CA (via secure API or admin approval workflow)
- CA signs it with a time-limited validity:
ssh-keygen -s /path/to/ca/ssh_ca -I user@example.com -n username -V +52w ~/.ssh/id_rsa.pub - User receives a
~/.ssh/id_rsa-cert.pubvalid for 52 weeks - User connects:
ssh -i ~/.ssh/id_rsa user@production-server - Server Configuration: Add to
/etc/ssh/sshd_configon all 1000 servers:
2️⃣
Remediation & Permanent Safeguards
Architecture: Operational Scale: With 1000 servers, update /etc/ssh/user_ca.pub once on all servers (via configuration management like Ansible), and cert-based auth works network-wide without touching private keys.
ssh-keygen -t rsa -f /path/to/ca/ssh_ca
- Benefits:
- No password distribution or memorization
- Automatic expiration (no need to revoke individual keys; certificates naturally expire)
- Centralized revocation list (CRL) if needed
- Audit trail: Each certificate embeds username, timestamp, and approval details
- Zero-trust: Revoke CA instantly, all certificates become invalid without manual key cleanup
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: CA Setup: Designate a secure certificate authority machine that signs user and host SSH keys. Generate a CA key pair:."
⚡ 60-Second Elevator Pitch Talking Points
- CA Setup: Designate a secure certificate authority machine that signs user and host SSH keys. Gen...
- User Authentication: Instead of distributing individual keys, users request a signed certificate:
- User generates their personal key: ssh-keygen -t rsa -f ~/.ssh/id_rsa
Advertisement