Q: Route 53 health check is failing for your endpoint but the endpoint seems fine.
When a Route 53 health check is failing while the endpoint looks fine from your laptop, the usual reason is that the checkers cannot reach or satisfy it the way you do: their source IPs are blocked by a security group, NACL or.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a <strong>Route 53 health check is failing</strong> while the endpoint looks fine from your laptop, the usual reason is that the checkers cannot reach or satisfy it the way you do: their source IPs are blocked by a security group, NACL or WAF, the Host header or SNI is wrong, the response is too slow or the string match fails. Route 53 health checkers run from multiple AWS regions, so your own browser test proves little. Interviewers want a diagnostic runbook: read the per-region status, reproduce the probe exactly, fix the blocking layer and prevent a false failover.
Read the evidence: per-region checker status
Start with what Route 53 itself reports instead of guessing.
- Console and CLI:
aws route53 get-health-check-status --health-check-id <id>lists each checker region with a status report such as a connection timeout or HTTP status. If only some regions fail, suspect geo or IP filtering. - Review the config:
aws route53 get-health-check --health-check-id <id>shows type, port, path, FullyQualifiedDomainName, EnableSNI, SearchString, RequestInterval and FailureThreshold. - CloudWatch: the
HealthCheckStatusmetric lives inus-east-1; create alarms there regardless of where the endpoint runs. - Remember the semantics: the endpoint is healthy only if enough checkers (default: more than 18% of them) report it healthy, and it flips after the failure threshold of consecutive failed checks.
- Interpret the pattern: all regions failing with timeouts points to an unreachable or slow endpoint; mixed results suggest partial blocking or a flapping backend; HTTP 4xx from every region suggests the path or Host header.
Hands-on: Route 53 health check failing, reproduce the probe
Replicate what the checker sends rather than what your browser sends.
# Reproduce as the checker would (replace host/path)
curl -sv -o /dev/null -w '%{http_code} connect=%{time_connect}s total=%{time_total}s\n' \
-H 'Host: app.example.com' --resolve app.example.com:443:203.0.113.10 \
https://app.example.com/health
aws route53 get-health-check-status --health-check-id abcdef11-2222-3333-4444-555555fedcba \
--query 'HealthCheckObservations[].[Region,StatusReport.Status]' --output table
- Host header and SNI: for HTTPS the checker sends the configured domain name or the IP-with-host-header combination; if EnableSNI is off, a multi-cert or ALB endpoint may return a default certificate or error.
- Path and status: the endpoint must return 2xx or 3xx within a few seconds. A login redirect chain, 401 or 403 on
/healthcounts as failure. - String matching: Route 53 searches only the first 5,120 bytes of the body, so a marker placed late in a large page is never seen.
- Speed: connection and response have short fixed timeouts; a slow health endpoint that queries a database often fails intermittently.
- Redirects: do not rely on the checker following a redirect to a healthy page; point it at a direct 200 path.
Allow the Route 53 health checker IP ranges
Public AWS publishes the source addresses; your firewall layers must permit them.
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json \
| jq -r '.prefixes[] | select(.service=="ROUTE53_HEALTHCHECKS") | .ip_prefix' | sort -u
# Example: allow 443 from one range (repeat per CIDR, or use a managed prefix list)
aws ec2 authorize-security-group-ingress --group-id sg-0abc1234 \
--protocol tcp --port 443 --cidr 15.177.0.0/18 # use a real CIDR from the list above
- Fetch the ranges: the
ROUTE53_HEALTHCHECKSservice inip-ranges.jsonlists the CIDRs. - Every layer: check the security group, subnet NACL, host firewall (iptables, ufw), on-prem firewall, WAF rules and rate limits. Bot-protection rules commonly block the checkers.
- Right port and protocol: a TCP check on 443 differs from an HTTPS check; mismatch with the listener is a classic mistake.
- Keep it current: subscribe to the AWS IP range change notification SNS topic or generate rules from the JSON in your pipeline.
- Verify after the change: wait a few intervals (default 30 seconds each, 10 seconds for fast checks) and re-run get-health-check-status; every region should flip to Success.
Root causes and special cases
Tie observed symptoms to a fix.
- Private endpoint: health checkers cannot reach private IPs or private hosted zone targets. Use a CloudWatch-alarm-based health check driven by metrics from inside the VPC.
- Wrong health path: checking
/which redirects to a login, or a path that exists only on one backend. - Target group vs DNS name: the check hits the ALB DNS name, whose backend instances may be unhealthy; evaluate target health separately.
- Fail-open behavior: if all records in a set are unhealthy Route 53 returns them all, so you may not see an outage even though checks are red.
- Calculated or inverted checks: verify no inversion flag or parent aggregate check explains the status.
- Cert problems: an expired or mismatched certificate fails HTTPS checks even though browsers on your machine may have cached exceptions.
Prevention and alerting
Make health checks reliable and observable.
- Dedicated health endpoint: fast, unauthenticated, shallow (process alive and critical dependency reachable) and under the timeout.
- Alarm on the metric: CloudWatch alarm on
HealthCheckStatus < 1notifies via SNS before DNS failover surprises anyone. - Define in IaC: manage
aws_route53_health_checkwith Terraform alongside the firewall rules so port and path stay in sync. - Test failover: deliberately fail the endpoint in staging to confirm routing policy and TTL behave as expected.
- Document ownership: record who owns the health path and what 'healthy' means so application changes do not silently break DNS failover.
- Route 53 health checkers probe from many AWS regions, so a healthy endpoint for me can still fail for them.
- I verify with get-health-check-status per region, then reproduce the probe with curl using the right Host header, SNI and path.
- I automate the checker IP allowlist from ip-ranges.json, define checks in Terraform and alarm on HealthCheckStatus in us-east-1.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Declaring the endpoint healthy because it loads in a browser The checkers come from other IPs and regions and may be blocked by security groups, NACLs or WAF.
- Pointing a public health check at a private IP Route 53 checkers are on the internet and cannot reach private addresses; use a CloudWatch alarm check instead.
- Putting the match string deep in a large response Only the first 5,120 bytes of the body are searched.
- Making the health endpoint call slow dependencies Timeouts are short, so the check flaps and can trigger needless failover.