Running an EKS Auto Mode cluster in a VPC with no direct internet connection.

1

Hi, I would like to know if it is possible to run an EKS Auto Mode cluster in a VPC that does not a direct internet connection, no NAT GW and IGW. I could not find a way to do it with a custom Node Class https://docs.aws.amazon.com/eks/latest/userguide/create-node-class.html In our environment we have a http proxy that is in another AWS account and peered VPC. I tried to instantiate an EKS auto mode cluster but no pods could be scheduled since the nodes could not start (I saw it through the Karpenter node claims). I suspect that is related to the fact that the kubelet on the AWS managed EC2 cannot reach the API server of the control plane that runs in an AWS managed VPC.

We used to have a cluster with a mix of Fargate nodes (core DNS + Karpenter) and Karpenter provisioned nodes and we had to configure bottlerocket userdata in a Node Class with specific network configuration: HTTPS_PROXY, NOPROXY environment to make nodes start properly. But with Auto Mode we can't control the userData. Is there any documentation about the AWS services that the EC2 instances need to connect to and how the kubelet accesses the API server ? (We can also add VPC endpoint interfaces in the peered VPC and route to them with route53 and routes in the route tables).

asked a month ago45 views
1 Answer
0

Greeting

Hi Luc,

Thanks for reaching out with such a detailed question! It sounds like you’re navigating a challenging setup with EKS Auto Mode in an environment without direct internet access. This is an edge case, and it’s great that you’re tackling it head-on. Let’s unpack this together and work toward a solution. 😊


Clarifying the Issue

You’re trying to run an EKS Auto Mode cluster in a VPC without direct internet connectivity—no Internet Gateway (IGW) or NAT Gateway (NAT GW). You’re using a custom HTTP proxy in another peered VPC, but the nodes in the cluster fail to start, likely because the kubelet on the managed EC2 instances cannot access the API server in AWS's managed VPC.

Previously, you successfully handled this setup using Fargate and Bottlerocket by configuring the necessary HTTPS_PROXY and NOPROXY settings. However, in Auto Mode, you cannot modify the user data for managed nodes, which has limited your control. This restriction is due to the managed nature of EKS Auto Mode, which abstracts infrastructure-level configurations to simplify Kubernetes operations.

Additionally, you’re seeking clarity on which AWS services and endpoints are critical for the EC2 instances to function correctly.


Why This Matters

In environments with strict security requirements or no direct internet access, achieving a functional and secure EKS cluster is critical for production workloads. Properly configuring a setup like this ensures compliance with organizational policies while enabling seamless communication between EKS components. However, the added complexity requires careful planning and robust monitoring to avoid operational bottlenecks.


Key Terms

  • EKS Auto Mode: A simplified configuration of Amazon EKS where AWS manages the cluster and its infrastructure, including worker nodes.
  • Kubelet: A Kubernetes component that runs on nodes to ensure containers are running and to communicate with the control plane.
  • HTTPS_PROXY/NOPROXY: Environment variables used to define proxy settings for outbound traffic.
  • VPC Peering: A networking connection between two VPCs that allows them to route traffic between each other privately.
  • VPC Endpoints: Private endpoints in a VPC that allow you to connect to AWS services without using an Internet Gateway.
  • Route 53 Private Hosted Zones: A DNS configuration enabling private domain resolution within a specific VPC or set of VPCs.

The Solution (Our Recipe)

Steps at a Glance:

  1. Identify AWS services and endpoints required for the EKS cluster.
  2. Configure VPC endpoints for these services in the VPC without direct internet access.
  3. Set up routing between the VPCs using VPC peering and Route 53.
  4. Configure DNS resolution to ensure nodes can resolve and access required endpoints.
  5. Configure proxy settings if required.
  6. Test and validate the cluster setup.

Step-by-Step Guide:

  1. Identify AWS Service Endpoints
    The managed EC2 instances need access to:
    • EKS Control Plane: For kubelet communication with the API server.
    • Container Registry (ECR): To pull container images.
    • CloudWatch Logs: For logging (optional but recommended).
    • S3: For AWS service metadata and configurations.
    Use the AWS VPC Endpoints documentation to list specific endpoints you’ll need to create.

  1. Create VPC Endpoints in the Isolated VPC
    In your VPC without internet access, create interface endpoints for:
    • com.amazonaws.region.eks
    • com.amazonaws.region.ecr.api
    • com.amazonaws.region.ecr.dkr
    • com.amazonaws.region.logs
    • com.amazonaws.region.s3
    Example using AWS CLI:
    aws ec2 create-vpc-endpoint --vpc-id vpc-0123456789abcdef \
        --service-name com.amazonaws.us-east-1.eks \
        --vpc-endpoint-type Interface

  1. Set Up Routing with VPC Peering

    • Ensure routes exist between the peered VPCs to allow traffic from the isolated VPC to the HTTP proxy and other required endpoints.
    • Update route tables to include the HTTP proxy and DNS resolver locations.
  2. Configure DNS Resolution

    • Use Route 53 Private Hosted Zones to map service endpoints to the VPC interface endpoints.
    • If the HTTP proxy requires custom DNS, configure the resolver to forward requests for specific domains to the proxy.

    Example DNS Resolution Test:

    dig eks.region.amazonaws.com

    Ensure the output resolves to the private IP address of the VPC endpoint. If the resolution fails, verify that the private hosted zone is correctly associated with your VPC and the appropriate Route 53 records are in place.


  1. Configure Proxy Settings if Required
    If additional proxy settings are needed, configure these at the VPC or endpoint level as applicable. For example:
    export HTTPS_PROXY=http://proxy-server:port
    export NOPROXY=127.0.0.1,localhost,.yourdomain.com
    Ensure these settings are propagated appropriately to the nodes. Validate them by checking that outbound traffic is routed correctly through the proxy.

  1. Validate Node Connectivity

    • Use tools like curl or wget on the managed EC2 instances to verify that they can reach required endpoints through the proxy.
    • Check for successful node registration in the Kubernetes API server.

    Example Node Connectivity Test:

    curl -x http://proxy-server:port https://eks.region.amazonaws.com/cluster-name

    A successful test will return an HTTP status code or the expected API server response. If this fails, examine the proxy configuration and VPC route tables for errors.


Closing Thoughts

By creating the necessary VPC endpoints, configuring routing, and ensuring proper DNS resolution, you can enable an EKS Auto Mode cluster to operate in a VPC without direct internet access. While this setup introduces additional complexity, it aligns with strict security requirements and maintains functionality.

This approach provides a secure, compliant method for deploying EKS Auto Mode clusters in environments with stringent networking restrictions. However, it requires robust planning, ongoing monitoring, and knowledge of AWS networking tools to maintain operational efficiency.

Here are some resources that might be helpful:


Farewell

I hope this helps you get your EKS Auto Mode cluster up and running in your secure environment, Luc! If you encounter any further issues or need clarification, feel free to ask. Wishing you success with your EKS deployment! 🎉😊


Cheers,

Aaron 😊

profile picture
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions