AWS EKS node groups: Code=NodeCreationFailure, Message=Instances failed to join the Kubernetes cluster

Question

The EKS node groups failed to join Kubernetes with this error message "NodeCreationFailure, Message=Instances failed to join the Kubernetes cluster". Below are my settings:
* Node group is created in private subnet.
* The EKS's Cluster end point access is Private.
* The VPC's configuration:
  * DNS settings, Enable DNS resolution and DNS hostnames are enabled.
  * DHCP option set has domain-nameservers equals to 169.254.169.254.

I have tried below configurations. But they didn't help.
* Enable public and private end point access.
* Add route to NAT gateway in route table associating with subnet.
* Use AWSSupport-TroubleshootEKSWorkerNode and get this error and warnings.

```
HERE IS A SUMMARY OF THE ITEMS THAT REQUIRE YOUR ATTENTION:

[ERROR]: AmazonProvidedDNS is not enabled
[WARNING]: Worker node outbound IP to internet is None. It is not allowed in the cluster Public CIDR ranges. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
[WARNING]: The default VPC Endpoint Policy is not being used. Please ensure that the worker node is able to communicate with the VPC Endpoint.
[WARNING]: The default VPC Endpoint Policy is not being used. Please ensure that the worker node is able to communicate with the VPC Endpoint.
[WARNING]: The default VPC Endpoint Policy is not being used. Please ensure that the worker node is able to communicate with the VPC Endpoint.
[WARNING]: The default VPC Endpoint Policy is not being used. Please ensure that the worker node is able to communicate with the VPC Endpoint.
[WARNING]: No secondary private IP addresses are assigned to worker node i-015c6dc20d04e1160, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
[WARNING]: The Outbound security group rules for worker node i-015c6dc20d04e1160 are customized and might limit outbound trafficto the EKS cluster endpoint. Please ensure that the security group outbound rules are matching the minimum requirements following this document: https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html
[WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status ".

============================================================================================================================================
Here are the detailed steps of the document execution:

[X] Checking EKS cluster EnterpriseEksCluster-Q6kOY4B4gjoY: EKS Cluster: EnterpriseEksCluster-Q6kOY4B4gjoY is in Active state.
    
1. Checking if the cluster Security Group is allowing traffic from the worker node:
       Passed: The cluster Security Group sg-07e405d757b121ae2 is allowing traffic from the worker node.
    
2. Checking DHCP options of the cluster VPC:
       [ERROR]: AmazonProvidedDNS is not enabled
:
```

Answer

Greetings,

I understand that you are facing error in node group creation as the instances failed to join the Kubernetes cluster. I can see that your EKS Cluster end point access is Private. Please allow me to mention that when you enable endpoint private access for your cluster, Amazon EKS creates a Route 53 private hosted zone on your behalf and associates it with your cluster's VPC. This private hosted zone is managed by Amazon EKS, and it doesn't appear in your account's Route 53 resources. In order for the private hosted zone to properly route traffic to your API server, your VPC must have enableDnsHostnames and enableDnsSupport set to true, and the DHCP options set for your VPC must include AmazonProvidedDNS in its domain name servers list.

Here are the steps to include the above-

1. Naviate to VPC in Console.
2. Find "DHCP option sets" under "Virtual private cloud" on the left.
3. You should be able to see your existing DHCP option set in the list. Unfortunately, we are not able to modify an existing one, so we create a new one.
4. Click "Create DHCP option set", and put everything as same as the existing one, EXCEPT, add "AmazonProvidedDNS" in "Domain name servers" configuration. It should be something like "X.X.X.X, X.X.X.X, AmazonProvidedDNS".
5. After you create it, go to your VPC and "Edit DHCP option set". Then choose the new DHCP option set.

Kindly refer the documentation [1] and [2] for more details.

Have a nice day!

References:

[1] https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
[2] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating

AWS EKS node groups: Code=NodeCreationFailure, Message=Instances failed to join the Kubernetes cluster

Relevant content