Skip to content

My Route 53 health check for an internal application is always unhealthy. How do I health check and fail over private resources?

0

I created a Route 53 health check pointing at the private IP address of an instance in my VPC so I could use a failover routing policy in my private hosted zone. The health check never becomes healthy. I have confirmed the application is running and reachable from inside the VPC. What am I doing wrong?

asked a month ago83 views

2 Answers
0
Accepted Answer

Nothing is wrong with your application — the health check simply cannot reach it. Route 53 health checkers are outside your VPC. They are public and can only monitor endpoints with IP addresses that are publicly routable on the internet. A private IP address is unreachable to them, so the check fails permanently.

Note that this is a limitation of the health checker, not of private hosted zones: you can associate health checks with failover records in a private hosted zone. You just need a health check that does not require public reachability.

Option 1 (recommended for private resources): a CloudWatch alarm-based health check

Instead of having Route 53 probe the endpoint, have it watch a CloudWatch alarm:

  1. Pick or publish a CloudWatch metric that reflects your application's health — for example the EC2 StatusCheckFailed metric, an ELB HealthyHostCount, or a custom metric your application publishes.
  2. Create a CloudWatch alarm on that metric.
  3. Create a Route 53 health check of type "state of a CloudWatch alarm."
  4. Associate that health check with your failover record.

Because the metric is produced inside your account rather than probed from the internet, this works for entirely private resources.

Option 2: expose a public probe target (only if appropriate)

If you can give the endpoint a public IP address, a standard Route 53 health check works. If you do this, restrict exposure:

  • Allow only the published Route 53 health checker IP ranges in your security groups and network ACLs. Look up the service ROUTE53_HEALTHCHECKS in the AWS IP address ranges list.
  • Be clear that this puts an internet-facing surface on an otherwise internal resource — for many internal applications Option 1 is the better trade-off.

Option 3: probe a public dependency instead

You can point a health check at an external resource that the instance relies on — for example a public database endpoint or an upstream API — rather than the private instance itself. This is an indirect signal, so use it only when the dependency's health genuinely represents your application's health.

Option 4: a Lambda-based probe (for TCP/HTTP/HTTPS checks inside the VPC)

AWS publishes a CloudFormation-based pattern that runs a Lambda function inside your VPC to perform TCP, HTTP, and HTTPS checks against private resources, publish the result to CloudWatch, and expose it as a Route 53 health check. Use this when you need an actual application-layer probe rather than an infrastructure metric. See the blog reference below.

Combining signals: calculated health checks

Route 53 also supports calculated health checks, which derive their status from other health checks. A common pattern is one alarm-based child health check per backend (for example HealthyHostCount >= 1 per load balancer) combined into a single parent health check that the DNS record references. This keeps the record simple while the health logic stays composable.

AWS

answered a month ago

EXPERT

reviewed a month ago

0

The issue you're experiencing is expected behavior. Route 53 health checkers operate outside of your VPC, which means they cannot directly reach resources that only have private IP addresses. When you create a health check pointing to a private IP address, the health checkers have no route to that endpoint, so the check will always fail.

To health check and fail over private resources in a VPC, you have two options:

  1. Assign a public IP address: If you assign a public IP address to your instance, Route 53 health checkers will be able to reach it directly. However, this may not be desirable for internal applications.

  2. Use CloudWatch metric-based health checks: This is the recommended approach for private resources. Instead of having Route 53 directly check your private endpoint, you can create a CloudWatch metric that monitors your resource (for example, using the EC2 StatusCheckFailed metric), associate a CloudWatch alarm with that metric, and then create a Route 53 health check based on the alarm's data stream. This way, the health check evaluates the CloudWatch alarm state rather than trying to reach the private IP directly. If the alarm state is OK, the health check is considered healthy; if the state is ALARM, it's considered unhealthy.

This CloudWatch-based approach allows you to monitor instances within your VPC that only have private IP addresses while still enabling failover functionality in your private hosted zone.
Sources
Configuring failover in a private hosted zone - Amazon Route 53
HealthCheckConfig - Amazon Route 53

answered a month ago

AWS
SUPPORT ENGINEER

reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.