- Newest
- Most votes
- Most comments
Greeting
Hi Luc,
Thanks for reaching out with such a detailed question! It sounds like you’re navigating a challenging setup with EKS Auto Mode in an environment without direct internet access. This is an edge case, and it’s great that you’re tackling it head-on. Let’s unpack this together and work toward a solution. 😊
Clarifying the Issue
You’re trying to run an EKS Auto Mode cluster in a VPC without direct internet connectivity—no Internet Gateway (IGW) or NAT Gateway (NAT GW). You’re using a custom HTTP proxy in another peered VPC, but the nodes in the cluster fail to start, likely because the kubelet on the managed EC2 instances cannot access the API server in AWS's managed VPC.
Previously, you successfully handled this setup using Fargate and Bottlerocket by configuring the necessary HTTPS_PROXY
and NOPROXY
settings. However, in Auto Mode, you cannot modify the user data for managed nodes, which has limited your control. This restriction is due to the managed nature of EKS Auto Mode, which abstracts infrastructure-level configurations to simplify Kubernetes operations.
Additionally, you’re seeking clarity on which AWS services and endpoints are critical for the EC2 instances to function correctly.
Why This Matters
In environments with strict security requirements or no direct internet access, achieving a functional and secure EKS cluster is critical for production workloads. Properly configuring a setup like this ensures compliance with organizational policies while enabling seamless communication between EKS components. However, the added complexity requires careful planning and robust monitoring to avoid operational bottlenecks.
Key Terms
- EKS Auto Mode: A simplified configuration of Amazon EKS where AWS manages the cluster and its infrastructure, including worker nodes.
- Kubelet: A Kubernetes component that runs on nodes to ensure containers are running and to communicate with the control plane.
- HTTPS_PROXY/NOPROXY: Environment variables used to define proxy settings for outbound traffic.
- VPC Peering: A networking connection between two VPCs that allows them to route traffic between each other privately.
- VPC Endpoints: Private endpoints in a VPC that allow you to connect to AWS services without using an Internet Gateway.
- Route 53 Private Hosted Zones: A DNS configuration enabling private domain resolution within a specific VPC or set of VPCs.
The Solution (Our Recipe)
Steps at a Glance:
- Identify AWS services and endpoints required for the EKS cluster.
- Configure VPC endpoints for these services in the VPC without direct internet access.
- Set up routing between the VPCs using VPC peering and Route 53.
- Configure DNS resolution to ensure nodes can resolve and access required endpoints.
- Configure proxy settings if required.
- Test and validate the cluster setup.
Step-by-Step Guide:
- Identify AWS Service Endpoints
The managed EC2 instances need access to:- EKS Control Plane: For kubelet communication with the API server.
- Container Registry (ECR): To pull container images.
- CloudWatch Logs: For logging (optional but recommended).
- S3: For AWS service metadata and configurations.
- Create VPC Endpoints in the Isolated VPC
In your VPC without internet access, create interface endpoints for:com.amazonaws.region.eks
com.amazonaws.region.ecr.api
com.amazonaws.region.ecr.dkr
com.amazonaws.region.logs
com.amazonaws.region.s3
aws ec2 create-vpc-endpoint --vpc-id vpc-0123456789abcdef \ --service-name com.amazonaws.us-east-1.eks \ --vpc-endpoint-type Interface
-
Set Up Routing with VPC Peering
- Ensure routes exist between the peered VPCs to allow traffic from the isolated VPC to the HTTP proxy and other required endpoints.
- Update route tables to include the HTTP proxy and DNS resolver locations.
-
Configure DNS Resolution
- Use Route 53 Private Hosted Zones to map service endpoints to the VPC interface endpoints.
- If the HTTP proxy requires custom DNS, configure the resolver to forward requests for specific domains to the proxy.
Example DNS Resolution Test:
dig eks.region.amazonaws.com
Ensure the output resolves to the private IP address of the VPC endpoint. If the resolution fails, verify that the private hosted zone is correctly associated with your VPC and the appropriate Route 53 records are in place.
- Configure Proxy Settings if Required
If additional proxy settings are needed, configure these at the VPC or endpoint level as applicable. For example:
Ensure these settings are propagated appropriately to the nodes. Validate them by checking that outbound traffic is routed correctly through the proxy.export HTTPS_PROXY=http://proxy-server:port export NOPROXY=127.0.0.1,localhost,.yourdomain.com
-
Validate Node Connectivity
- Use tools like
curl
orwget
on the managed EC2 instances to verify that they can reach required endpoints through the proxy. - Check for successful node registration in the Kubernetes API server.
Example Node Connectivity Test:
curl -x http://proxy-server:port https://eks.region.amazonaws.com/cluster-name
A successful test will return an HTTP status code or the expected API server response. If this fails, examine the proxy configuration and VPC route tables for errors.
- Use tools like
Closing Thoughts
By creating the necessary VPC endpoints, configuring routing, and ensuring proper DNS resolution, you can enable an EKS Auto Mode cluster to operate in a VPC without direct internet access. While this setup introduces additional complexity, it aligns with strict security requirements and maintains functionality.
This approach provides a secure, compliant method for deploying EKS Auto Mode clusters in environments with stringent networking restrictions. However, it requires robust planning, ongoing monitoring, and knowledge of AWS networking tools to maintain operational efficiency.
Here are some resources that might be helpful:
- EKS Cluster VPC Requirements
- VPC Endpoints for Amazon EKS
- Using HTTP Proxies with AWS Services
- Route 53 Private Hosted Zones
- Using ECR in Private VPCs
Farewell
I hope this helps you get your EKS Auto Mode cluster up and running in your secure environment, Luc! If you encounter any further issues or need clarification, feel free to ask. Wishing you success with your EKS deployment! 🎉😊
Cheers,
Aaron 😊
Relevant content
- asked 2 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 2 years ago
- How do I provide cluster access to other IAM users and roles after I create a cluster in Amazon EKS?AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 4 months ago