By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EKS worker nodes can not join my Amazon EKS cluster

0

Hi,

I have configured my EKS cluster as public + private cluster endpoint and tried with only private cluster endpoint. From a VPC configuration perspective, my worker nodes are deployed in a private subnet. Looking at AWS documentation, the following are stated:

https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html If the node is deployed to a private subnet, then the subnet must have a route to a NAT gateway that has a public IP address assigned to it.

https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html API server endpoint access options Endpoint public access = Enabled Endpoint private access = Enabled Behavior: Kubernetes API requests within your cluster's VPC (such as node to control plane communication) use the private VPC endpoint. Your cluster API server is accessible from the internet. You can, optionally, limit the CIDR blocks that can access the public endpoint.

If the setup where I have both Endpoint public and private accesses enabled, why should I have a NAT gateway configured for the node be able to join the cluster? Above the behavior described says that node to control plane communication will use private VPC endpoint (ENIs that EKS Control Plane master node provsion in the associated VPC). When I add the NAT, the node is able to join the cluster, but it shouldnt be the case right? I would rather use NAT to reach other services such as ECR, but not expected to communicate with EKS control plane via NAT.

Please support.

Topics
Containers
Tags
Amazon Elastic Kubernetes Service
Language
English
Fabio R
asked a year ago3.2K views
2 Answers
1
Accepted Answer

Hi Fabio curious which VPC Endpoints did you create in this private subnet, did you by chance create one for EC2 api? This might be what you are missing for your nodes to join. (if you have ECR, STS, S3 plus anything else you might need ALB, CW etc but not necessary for nodes). https://repost.aws/knowledge-center/resolve-eks-node-failures

AWS
AWS-Aaron-S
answered a year ago
profile picture
EXPERT
Oleksii Bebych
reviewed 7 months ago
profile picture
EXPERT
Giovanni Lauria
reviewed 10 months ago
  • Fabio R
    a year ago

    You are correct. EC2 API end-point was missing indeed.

0

With Public and Private Endpoints are enabled then yes nodes in Private Subnet should be able to join cluster without traversing the Public internet.

Sounds like you might be missing these setting in your VPC, if you are properly enabling both Public and Private Endpoints.

When creating the Endpoints EKS creates a Private Hosted Zone (which will allow your nodes to look up Public Endpoint name as it doesnt change and resolve it to private VPC Endpoints managed by EKS). This private hosted zone is managed by Amazon EKS, and it doesn't appear in your account's Route 53 resources. **In order for the private hosted zone to properly route traffic to your API server, your VPC must have enableDnsHostnames and enableDnsSupport set to true, and the DHCP options set for your VPC must include AmazonProvidedDNS in its domain name servers list. **

https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html

AWS
AWS-Aaron-S
answered a year ago
  • Fabio R
    a year ago

    All described settings were in place, yet I still need the NAT in order to register my node located at a private network to a cluster endpoint enabled with both private and public access. The endpoint FQDN indeed return when issued inside the VPC the up addresses of the EKS-managed ENIs. Certainly I will require a NAT gateway to access AWS ECR for instance, or created a VPC interface endpoint, but it is annoying me why I can’t register the node with NAT. I have ran the EKS node automated troubleshooting tool provided by AWS, and it still indicates the absence of a NAT GW.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content