How do I troubleshoot DNS resolution issues in my Route 53 private hosted zone?

Short description

To fix private hosted zone DNS issues, verify Amazon Virtual Private Cloud (Amazon VPC) settings, zone associations, and DNS server setup.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Confirm DNS support in the VPC

To allow private hosted zone record resolution, complete the following steps:

1. Turn on DNS support in your Amazon VPC.
2. Verify that DNSSupport and DNSHostnames are set to True in your VPC. For more information, see View and update DNS attributes for your VPC.

Confirm the correct VPC ID association

Confirm that you associated the correct VPC ID with your private hosted zone. Also, make sure that you're querying the resource records of the domain from within the same VPC.

When you associate a private hosted zone with your VPC:

• Route 53 Resolver creates an auto-defined rule and associates it with your VPC.
• Resources in your VPC can query the Resolver to resolve DNS records in the private hosted zone.

To list VPCs that are associated with a hosted zone, run the following command in the AWS CLI:

aws route53 get-hosted-zone --id VPC_ID

Note: Replace VPC_ID with your relevant values.

To list private hosted zones that are associated with specific VPCs, run the following command in the AWS CLI:

aws route53 list-hosted-zones-by-vpc --vpc-id VPC_ID --vpc-region REGION_ID

Note: Replace HOSTED_ZONE_ID, VPC_ID, and REGION_ID with your relevant values.

Check custom DNS server configurations

If you configured custom DNS servers or Active Directory servers in the DHCP options for DNS in your VPC, then verify the following items:

1. Forwarding rule: The servers forward private domain DNS queries to your VPC's DNS server IP address.
2. Domain configuration: The domain in custom servers differs from your private hosted zone.

For example, if the primary CIDR range for your VPC is 172.31.0.0/16, then the VPC DNS server's IP address is 172.31.0.2. This is the Amazon VPC network range plus two.

Review Resolver configuration settings

If you experience intermittent DNS resolution or responses, then review the Resolver configuration settings of your source instance:

• For Linux instances, use the cat /etc/resolv.conf and cat/etc/hosts files.
• For macOS, see Change DNS Settings on Mac in the macOS user guide.
• For Windows, complete the following steps:
  Choose Settings, and then select Network & internet.
  Under Advanced network settings, select Change adapter settings.
  Right-click on the network connection, and then select Properties.
  Choose IPv4 properties, and then enter the preferred DNS IP address in DNS server addresses.

For example, if you configured resolv.conf, then you can use the rotate option to load balance queries between Amazon DNS and Google DNS (8.8.8.8). The resolv.conf file would look similar to the following:

options rotate; generated by /usr/sbin/dhclient-script
nameserver 8.8.8.8
nameserver 172.31.0.2

In your first query to the public Google DNS (8.8.8.8), you receive the expected NXdomain response. The Resolver is trying to find the response in the public hosted zone instead of the private hosted zone:

Private hosted Zone Record - resolvconf.local
[ec2-user@ip-172-31-253-89 etc]$ curl -vks http://resolvconf.local* Rebuilt URL to: http://resolvconf.local/
* Could not resolve host: resolvconf.local

15:24:58.553320 IP ip-172-31-253-89.ap-southeast-2.compute.internal.40043 > dns.google.domain: 65053+ A? resolvconf.local. (34)
15:24:58.554814 IP dns.google.domain > ip-172-31-253-89.ap-southeast-2.compute.internal.40043: 65053 NXDomain 0/1/0 (109)

However, the second query resolves successfully. The second query reaches the VPC DNS resolver that's associated to your private hosted zone:

[ec2-user@ip-172-31-253-89 etc]$ curl -vks http://resolvconf.local* Rebuilt URL to: http://resolvconf.local/*   Trying 1.1.1.1...
* TCP_NODELAY set
* Connected to resolvconf.local (1.1.1.1) port 80 (#0)

15:25:00.224761 IP ip-172-31-253-89.ap-southeast-2.compute.internal.51578 > 172.31.0.2.domain: 7806+ A? resolvconf.local. (34)
15:25:00.226527 IP 172.31.0.2.domain > ip-172-31-253-89.ap-southeast-2.compute.internal.51578: 7806 1/0/0 A 1.1.1.1 (50)

Confirm that private hosted zones don't have overlapping namespaces

When multiple zones have overlapping namespaces:

• Resolver routes traffic based on the most specific match
• If a matching zone exists but no matching record, then Resolver returns NXDOMAIN

Confirm that you configured the correct record in the most specific private hosted zone for successful DNS resolution.

For example, if you have two private hosted zones with the following records:

Private hosted zone	Record name	Value
local	overlap.privatevpc.local	60.1.1.1
privatevpc.local	overlap.privatevpc.local	50.1.1.1

Then, you receive the following query result from the most specific matched private hosted zone:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig overlap.privatevpc.local +short 
50.1.1.1

Check for zone/subdomain delegation in private hosted zones

Private hosted zones don't support zone/subdomain delegation. Confirm that you don't have a name server (NS) record configured for the subdomain in the private hosted zone of the parent domain. If you configured delegation, then the client gets the "SERVFAIL" response code from the VPC resolver.

The following is a delegation configuration example that causes SERVFAIL:

• Private hosted zone: abc.com
• Delegation NS record: kc.abc.com
• Resource record: test.kc.abc.com

[ec2-user@ip-172-31-0-8 ~]$ dig test.kc.abc.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 63414
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;test.kc.abc.com        IN      A
;; Query time: 15 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Fri Apr 16 15:57:37 2021
;; MSG SIZE  rcvd: 48

Confirm routing policy support

Confirm that you configured a routing policy in your resource record that's supported by a private hosted zone. For more information, see Supported routing policies for records in a private hosted zone.

Check the Resolver rule and outbound Resolver endpoint usage

Make sure that you're using Resolver with an outbound endpoint. The Resolver rule takes precedence if:

1. You have a Resolver rule to route traffic to your network for your private hosted zone's domain.
2. You have a Resolver rule that's associated with the same VPC that's also associated with the private hosted zone.

For more information, see Resolving DNS queries between VPCs and your network.

Prevent query loops

To avoid creating a loop, complete the following steps:

1. Don't create target IP addresses in a Resolver forwarding rule that point to inbound endpoints of your VPC.
2. Don't associate the endpoints with the private hosted zone.
3. Don't associate the same Resolver rule with your VPC.

Example of a query loop:

ubuntu@ip-172-32-254-37:~$ dig overlap.privatevpc.local
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 9007
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;overlap.privatevpc.local. IN A
;; Query time: 2941 msec
;; SERVER: 172.32.0.2#53(172.32.0.2)

To resolve this issue and break the loop, remove the hub VPC association with the rule. Example of a successful response:

ubuntu@ip-172-32-254-37:~$ dig overlap.privatevpc.local
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58606
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;overlap.privatevpc.local. IN A
;; ANSWER SECTION:
overlap.privatevpc.local. 0 IN A 50.1.1.1
;; Query time: 5 msec
;; SERVER: 172.32.0.2#53(172.32.0.2)

Confirm that the on-premises Resolver sends recursive requests

For queries from on-premises to Route 53 Resolver:

• Use the Resolver inbound endpoint to forward DNS queries.
• Make sure that the on-premises Resolver sends recursive (not iterative) queries.

To verify resolution type, complete these steps:

1. Use packet capture on the on-premises DNS Resolver.
2. Review DNS flags (recursion desired = 0).
3. Test with +norecurse dig command, or set norecurse with nslookup.

Example of a failed iterative query:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig @172.31.253.150 overlap.privatevpc.local +norecurse
;; <<>> DiG 9.11.0rc1 <<>> @172.31.253.150 overlap.privatevpc.local +norecurse; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Example of a successful recursive query:

[ec2-user@IAD-BAS-INSTANCE ~]$ dig @172.31.253.150 overlap.privatevpc.local
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19051
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;overlap.privatevpc.local.      IN      A
;; ANSWER SECTION:
overlap.privatevpc.local. 0     IN      A       50.1.1.1
;; Query time: 200 msec
;; SERVER: 172.31.253.150#53(172.31.253.150)

Verify the correct rule priorities for Amazon provided DNS

When the client instance sends a query to the Resolver, the Resolver verifies the instance's rules for where to route the request.

The most specific rule takes priority. For example, if there's a test.example.com Resolver rule and a test.example.com private hosted zone, then the Resolver rule takes priority. The query forwards to the servers or target IP addresses that are configured in the rule.

If the rules are at the same domain level, then they have the following priorities:

1. Resolver rule
2. Private hosted zone rule
3. Internal rule

Related information

Working with private hosted zones

Which Amazon VPC options do I need to turn on to use my private hosted zone?

Avoid loop configurations with Resolver endpoints