⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 62 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE [L2] AWS Cost & Architecture Production Scenario [L2]
🎯 Target Role / Context: AWS Scenario #61 (L2) · Cost & Architecture

Q: Route 53 health check is failing for your endpoint but the endpoint seems fine.

When a Route 53 health check is failing while the endpoint looks fine from your laptop, the usual reason is that the checkers cannot reach or satisfy it the way you do: their source IPs are blocked by a security group, NACL or.

#AWS #Cost & Architecture #L2 #Cloud #Infrastructure
🎙️ Candidate Opening & Architectural Context
"When a <strong>Route 53 health check is failing</strong> while the endpoint looks fine from your laptop, the usual reason is that the checkers cannot reach or satisfy it the way you do: their source IPs are blocked by a security group, NACL or WAF, the Host header or SNI is wrong, the response is too slow or the string match fails. Route 53 health checkers run from multiple AWS regions, so your own browser test proves little. Interviewers want a diagnostic runbook: read the per-region status, reproduce the probe exactly, fix the blocking layer and prevent a false failover."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When a <strong>Route 53 health check is failing</strong> while the endpoint looks fine from your laptop, the usual reason is that the checkers cannot reach or satisfy it the way you do: their source IPs are blocked by a security group, NACL or WAF, the Host header or SNI is wrong, the response is too slow or the string match fails. Route 53 health checkers run from multiple AWS regions, so your own browser test proves little. Interviewers want a diagnostic runbook: read the per-region status, reproduce the probe exactly, fix the blocking layer and prevent a false failover.

1️⃣

Read the evidence: per-region checker status

Start with what Route 53 itself reports instead of guessing.

  • Console and CLI: aws route53 get-health-check-status --health-check-id <id> lists each checker region with a status report such as a connection timeout or HTTP status. If only some regions fail, suspect geo or IP filtering.
  • Review the config: aws route53 get-health-check --health-check-id <id> shows type, port, path, FullyQualifiedDomainName, EnableSNI, SearchString, RequestInterval and FailureThreshold.
  • CloudWatch: the HealthCheckStatus metric lives in us-east-1; create alarms there regardless of where the endpoint runs.
  • Remember the semantics: the endpoint is healthy only if enough checkers (default: more than 18% of them) report it healthy, and it flips after the failure threshold of consecutive failed checks.
  • Interpret the pattern: all regions failing with timeouts points to an unreachable or slow endpoint; mixed results suggest partial blocking or a flapping backend; HTTP 4xx from every region suggests the path or Host header.
2️⃣

Hands-on: Route 53 health check failing, reproduce the probe

Replicate what the checker sends rather than what your browser sends.

# Reproduce as the checker would (replace host/path)
curl -sv -o /dev/null -w '%{http_code} connect=%{time_connect}s total=%{time_total}s\n' \
  -H 'Host: app.example.com' --resolve app.example.com:443:203.0.113.10 \
  https://app.example.com/health

aws route53 get-health-check-status --health-check-id abcdef11-2222-3333-4444-555555fedcba \
  --query 'HealthCheckObservations[].[Region,StatusReport.Status]' --output table
  • Host header and SNI: for HTTPS the checker sends the configured domain name or the IP-with-host-header combination; if EnableSNI is off, a multi-cert or ALB endpoint may return a default certificate or error.
  • Path and status: the endpoint must return 2xx or 3xx within a few seconds. A login redirect chain, 401 or 403 on /health counts as failure.
  • String matching: Route 53 searches only the first 5,120 bytes of the body, so a marker placed late in a large page is never seen.
  • Speed: connection and response have short fixed timeouts; a slow health endpoint that queries a database often fails intermittently.
  • Redirects: do not rely on the checker following a redirect to a healthy page; point it at a direct 200 path.
Advertisement
3️⃣

Allow the Route 53 health checker IP ranges

Public AWS publishes the source addresses; your firewall layers must permit them.

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json \
 | jq -r '.prefixes[] | select(.service=="ROUTE53_HEALTHCHECKS") | .ip_prefix' | sort -u

# Example: allow 443 from one range (repeat per CIDR, or use a managed prefix list)
aws ec2 authorize-security-group-ingress --group-id sg-0abc1234 \
  --protocol tcp --port 443 --cidr 15.177.0.0/18   # use a real CIDR from the list above
  • Fetch the ranges: the ROUTE53_HEALTHCHECKS service in ip-ranges.json lists the CIDRs.
  • Every layer: check the security group, subnet NACL, host firewall (iptables, ufw), on-prem firewall, WAF rules and rate limits. Bot-protection rules commonly block the checkers.
  • Right port and protocol: a TCP check on 443 differs from an HTTPS check; mismatch with the listener is a classic mistake.
  • Keep it current: subscribe to the AWS IP range change notification SNS topic or generate rules from the JSON in your pipeline.
  • Verify after the change: wait a few intervals (default 30 seconds each, 10 seconds for fast checks) and re-run get-health-check-status; every region should flip to Success.
4️⃣

Root causes and special cases

Tie observed symptoms to a fix.

  • Private endpoint: health checkers cannot reach private IPs or private hosted zone targets. Use a CloudWatch-alarm-based health check driven by metrics from inside the VPC.
  • Wrong health path: checking / which redirects to a login, or a path that exists only on one backend.
  • Target group vs DNS name: the check hits the ALB DNS name, whose backend instances may be unhealthy; evaluate target health separately.
  • Fail-open behavior: if all records in a set are unhealthy Route 53 returns them all, so you may not see an outage even though checks are red.
  • Calculated or inverted checks: verify no inversion flag or parent aggregate check explains the status.
  • Cert problems: an expired or mismatched certificate fails HTTPS checks even though browsers on your machine may have cached exceptions.
5️⃣

Prevention and alerting

Make health checks reliable and observable.

  • Dedicated health endpoint: fast, unauthenticated, shallow (process alive and critical dependency reachable) and under the timeout.
  • Alarm on the metric: CloudWatch alarm on HealthCheckStatus < 1 notifies via SNS before DNS failover surprises anyone.
  • Define in IaC: manage aws_route53_health_check with Terraform alongside the firewall rules so port and path stay in sync.
  • Test failover: deliberately fail the endpoint in staging to confirm routing policy and TTL behave as expected.
  • Document ownership: record who owns the health path and what 'healthy' means so application changes do not silently break DNS failover.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Run aws route53 get-health-check-status first; if only some checker regions fail, you have a firewall, WAF or geo filter blocking the ROUTE53_HEALTHCHECKS ranges."
⚡ 60-Second Elevator Pitch Talking Points
  • Route 53 health checkers probe from many AWS regions, so a healthy endpoint for me can still fail for them.
  • I verify with get-health-check-status per region, then reproduce the probe with curl using the right Host header, SNI and path.
  • I automate the checker IP allowlist from ip-ranges.json, define checks in Terraform and alarm on HealthCheckStatus in us-east-1.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Declaring the endpoint healthy because it loads in a browser The checkers come from other IPs and regions and may be blocked by security groups, NACLs or WAF.
  2. Pointing a public health check at a private IP Route 53 checkers are on the internet and cannot reach private addresses; use a CloudWatch alarm check instead.
  3. Putting the match string deep in a large response Only the first 5,120 bytes of the body are searched.
  4. Making the health endpoint call slow dependencies Timeouts are short, so the check flaps and can trigger needless failover.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →