EKS Auto Mode cluster Can't Add Node

Question

Hi AWS Support Team,

We are experiencing a critical issue with our EKS Auto Mode cluster where worker nodes (including AL2023 and Bottlerocket) fail to join the cluster, getting stuck in 'NotReady' or failing to register entirely.

[Root Cause analyzed from kubelet logs]:
Our VPC uses a custom DHCP Option Set which enforces the traditional private DNS naming convention (e.g., ) on EC2 instances. 
When kubelet starts, it detects and registers using this FQDN hostname. However, EKS Auto Mode's managed network controller/control plane expects the node to identify itself strictly via its Instance ID ().

This leads to a fatal Node Restriction webhook rejection on the API Server side:
"is forbidden: User 'system:node: ' cannot get resource 'nodes' / 'csinodes' at the cluster scope: only its own Node object / can only access CSINode with the same name as the requesting node"

[Why we cannot bypass this via standard workarounds]:
1. EKS Auto Mode completely ignores the traditional 'aws-auth' ConfigMap.
2. In EKS Access Entries, selecting the 'EC2' type locks the username mapping to 'system:node:{{SessionName}}' and cannot be changed.
3. If we select 'Standard' type to customize the username mapping to 'system:node:{{EC2PrivateDNSName}}', the AWS Console blocks it with the error: "The username must not begin with system:".

Since this is an EKS Auto Mode architectural deadlock when interacting with a custom VPC DHCP environment, could you please internally review if the EKS control plane can loose the constraint or provide an official way to map custom hostnames for Auto Mode managed node groups?

Thank you.

jacksli · Accepted Answer

when i add InstanceIdNodeName: true to my lanuch template,this problem is resolved

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MIME_BOUNDARY"

--MIME_BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  featureGates:
    InstanceIdNodeName: true
--MIME_BOUNDARY--

Taka · Answer

I recommend to modify the VPC DHCP options set only for the node subnets.
Assign a dedicated DHCP options set to the subnets used for EKS worker nodes, reverting the private DNS naming convention to the AWS default (`ip-x-x-x-x.region.compute.internal`):

AWS Console

```
VPC → DHCP Options Sets → Create DHCP options set
→ Domain name: `ap-northeast-1.compute.internal` (for Tokyo)
→ Domain name servers: `AmazonProvidedDNS`

VPC → Select the VPC
→ "Actions" → "Edit DHCP options set"
→ Apply only to the VPC containing the worker node subnets
```

jacksli · Answer

i have check this network,it is ok.
i login my server and edit /etc/eks/kubelet/environment. when i change --hostname-override=`hostname` to --hostname-override=`instanceid`, all it works

Jul 09 02:00:06 ip-10-1-8-179.ec2.internal kubelet[2193]: E0709 02:00:06.484669    2193 kubelet.go:3526] "Unable to register mirror pod because node is not registered yet" err="node \"ip-10-1-8-179.ec2.internal\" not found" node="ip-10-1-8-179.ec2.internal"
Jul 09 02:00:06 ip-10-1-8-179.ec2.internal kubelet[2193]: I0709 02:00:06.520149    2193 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-1-8-179.ec2.internal" is forbidden: User "system:node:i-0b553101a7
b858a3f" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope: can only access CSINode with the same name as the requesting node

re:Post Agent · Answer

Thank you for the detailed analysis of your EKS Auto Mode node registration issue. Based on the symptoms you've described, there are several areas to investigate:

**Network Configuration Requirements**

EKS Auto Mode automatically configures EC2 instances with the correct information to join the cluster, including the cluster endpoint and cluster certificate authority. However, nodes can still fail to join due to network connectivity issues. When nodes are provisioned in private subnets without proper internet connectivity, they cannot communicate with the EKS API server, which prevents them from joining the cluster and can lead to multiple node creation attempts with pods remaining in pending state.

Ensure your private subnet's route table includes a default route pointing to a NAT Gateway to allow nodes to reach the EKS API server endpoint and external services.

**DNS and DHCP Configuration**

For standard EKS clusters, nodes require proper DNS resolution. The VPC where nodes are created should have values set for `domain-name` and `domain-name-servers` as Options in a DHCP options set. The default values are `domain-name:.compute.internal` and `domain-name-servers:AmazonProvidedDNS`. When custom DNS is used instead of AmazonProvidedDNS, worker nodes might not resolve the endpoint correctly, particularly when public access to the endpoint is disabled and only private access is enabled.

**Diagnosing Node Join Failures**

To identify instances that didn't join the cluster, run `kubectl get nodeclaim` to check for NodeClaims that show `Ready = False`. Then run `kubectl describe nodeclaim ` and look under the Status section to find any issues preventing the node from joining the cluster.

You can also retrieve node logs using the Kubernetes NodeDiagnostic resource or the AWS EC2 CLI command `get-console-output` to view troubleshooting and debugging information.

**Next Steps**

For architectural questions specific to EKS Auto Mode's interaction with custom DHCP configurations and node identity mapping, I recommend opening a support case through the AWS Support Center (https://console.aws.amazon.com/support/) where the support team can work directly with the EKS service team to investigate this specific scenario and provide guidance on supported configurations or potential workarounds.  
**Sources**  
[Troubleshoot problems with Amazon EKS clusters and nodes - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html)  
[Troubleshoot EKS Auto Mode - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/auto-troubleshoot.html)  
[How to Troubleshoot Cluster Scaling Issues in EKS Auto Mode | AWS re:Post](https://repost.aws/articles/ARLpQOknr5Rb-w5iAT9sUBpQ/how-to-troubleshoot-cluster-scaling-issues-in-eks-auto-mode)

EKS Auto Mode cluster Can't Add Node

Add your answer

Relevant content