How do I use multiple CIDR ranges with Amazon EKS?

Short description

To use multiple CIDR ranges in Amazon EKS, complete the following steps:

1. Add more CIDR ranges to expand your virtual private cloud (VPC) network.
2. Create subnets with a new CIDR range.
3. Associate your new subnet to a route table.
4. Configure the Amazon Virtual Private Cloud (Amazon VPC) Container Network Interface (CNI) plugin to use a new CIDR range.

Note: It's a best practice to use CIDRs (/16) from the carrier-grade NAT (CGN) space such as 100.64.0.0/10 or 198.19.0.0/16. You can use a valid VPC range as a secondary CIDR range in custom networks. However, these ranges are less likely to be used in a corporate setting. For more information about custom networking requirements, see Considerations.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Prerequisites: You must have the following configurations:

• A running Amazon EKS cluster
• AWS Identity and Access Management (IAM) permissions to manage an Amazon VPC
• A kubectl with permissions to create custom resources and edit the DaemonSet
• An installed version of jq on your system
  Note: To download and install jq, see Download jq on the jq website.
• A Unix-based system with a Bash shell
• A configured VPC

Add more CIDR ranges to expand your VPC network

Complete the following steps:

1. To retrieve the ID of your cluster VPC and store it in a variable, run the following AWS CLI describe-cluster command:

   VPC_ID=$(aws eks describe-cluster --name CLUSTER_NAME --region REGION_CODE --query "cluster.resourcesVpcConfig.vpcId" --output text)

   Note: Replace CLUSTER_NAME with the name of your cluster, and REGION_CODE with the AWS Region where you deployed the cluster.
2. To associate another CIDR block with the 100.64.0.0/16 range to the VPC, run the following associate-vpc-cidr-block command:

   aws ec2 associate-vpc-cidr-block --vpc-id $VPC_ID --cidr-block 100.64.0.0/16

Create subnets with a new CIDR range

Complete the following steps:

1. To list all the Availability Zones in your Region, run the following describe-availability-zones command:

   aws ec2 describe-availability-zones --region REGION_CODE --query 'AvailabilityZones[*].ZoneName'

   Note: Replace REGION_CODE with the Region where you deployed your resources.
2. Create the subnets that you want to use in each Availability Zone that your existing subnets are in. To create new subnets under the VPC with the new CIDR range, run the following create-subnet commands.

   CUST_SNET1=$(aws ec2 create-subnet --cidr-block 100.64.0.0/19 --vpc-id $VPC_ID --availability-zone AZ1 | jq -r .Subnet.SubnetId)
   CUST_SNET2=$(aws ec2 create-subnet --cidr-block 100.64.32.0/19 --vpc-id $VPC_ID --availability-zone AZ2 | jq -r .Subnet.SubnetId)
   CUST_SNET3=$(aws ec2 create-subnet --cidr-block 100.64.64.0/19 --vpc-id $VPC_ID --availability-zone AZ3 | jq -r .Subnet.SubnetId)

   Note: Replace AZ1, AZ2, and AZ3 with your existing subnet Availability Zones.
3. (Optional) To set a key-value pair to add a name tag to your subnets, run the following create-tags command:

   aws ec2 create-tags --resources $CUST_SNET1 --tags Key=KEY,Value=VALUE
   aws ec2 create-tags --resources $CUST_SNET2 --tags Key=KEY,Value=VALUE
   aws ec2 create-tags --resources $CUST_SNET3 --tags Key=KEY,Value=VALUE

   Note: Replace KEY with the tag key, and VALUE with the tag value.

Associate your new subnet to a route table

By default, Amazon VPC associates your new subnets with your VPC's main route table. This route table allows communication only between the resources that are deployed in the VPC. To allow communication with IP addresses that are outside the VPC's CIDR blocks, you must create your own route table for your subnets.

Configure the Amazon VPC CNI plugin to use the new CIDR range

Complete the following steps:

1. Make sure that you run the correct plugin version for your Kubernetes version. To verify the version, run the following command:

   kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

   If you don't have the correct plugin version, then update the Amazon VPC CNI.

2. To activate the custom network configuration for the Amazon VPC CNI plugin, run the following command:

   kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

3. To add the ENIConfig label to identify your worker nodes, run the following command:

   kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

4. To create ENIConfig custom resources for all subnets and Availability Zones, run the following commands:

   cat <<EOF  | kubectl apply -f -apiVersion: crd.k8s.amazonaws.com/v1alpha1
   kind: ENIConfig
   metadata:
    name: $AZ1
   spec:
     securityGroups:
       - cluster_security_group_id
     subnet: $CUST_SNET1
   EOF
   
   cat <<EOF | kubectl apply -f -
   apiVersion: crd.k8s.amazonaws.com/v1alpha1
   kind: ENIConfig
   metadata:
    name: $AZ2
   spec:
     securityGroups:
       - cluster_security_group_id
     subnet: $CUST_SNET2
   EOF
   
   cat <<EOF | kubectl apply -f -
   apiVersion: crd.k8s.amazonaws.com/v1alpha1
   kind: ENIConfig
   metadata:
    name: $AZ3
   spec:
     securityGroups:
       - cluster_security_group_id
     subnet: $CUST_SNET3
   EOF

   Note: Replace cluster_security_group_id with the ID of the existing security group that you want to use for each ENIConfig resource. The ENIConfig resource names are the same as the Availability Zone where you created the new subnets.

5. If you use a self-managed node group, then launch self-managed worker nodes to allow the plugin to allocate IP addresses to them. For Subnets, use the original subnets that you chose when you created the cluster instead of the new subnets that you used in the ENIConfig resources. You can also use a managed node group with a launch template to launch nodes. In both cases, you must identify the correct max-pods value for your instance type. This is because custom networking in Amazon EKS doesn't use the primary network interface for Pod placement. Make sure that you add the --cni-custom-networking-enabled parameter when you run the max-pods-calculator.sh script.
   Then, run the following command to specify the max-pods value for your self-managed nodes:

   --use-max-pods false --kubelet-extra-args '--max-pods=20'

   Note: Replace 20 with the max-pods value that you calculated.
   For a managed node group, add the following script to the User data:

   #!/bin/bash
   /etc/eks/bootstrap.sh my-cluster-name --use-max-pods false --kubelet-extra-args '--max-pods=20'

   Note: Replace my-cluster-name with your cluster name, and 20 with the max-pods value that you calculated. In your launch template, specify an Amazon EKS optimized Amazon Machine Image (AMI) ID, or a custom AMI built off the Amazon EKS optimized AMI. Amazon Linux 2023 (AL2023) AMIs use the nodeadm process. For information about how this changes the User data configuration, see Upgrade from Amazon Linux 2 to Amazon Linux 2023.
   If you use a managed node group without a launch template or specified AMI ID, then the managed node group automatically calculates the max-pods value.

6. After the new nodes are ready, drain the previous nodes to gracefully shut down the Pods. For more information, see Safely drain a node on the Kubernetes website. If the nodes are in an existing managed node group, then run the following command to delete the node group:

   aws eks delete-nodegroup --cluster-name CLUSTER_NAME --nodegroup-name my-nodegroup

   Note: Replace CLUSTER_NAME with your cluster name, and my-nodegroup with your node group name.

7. To launch a new deployment to test the configuration, run the following command:

   kubectl create deployment nginx-test --image=nginx --replicas=10   
   kubectl get pods -o wide --selector=app=nginx-test

   Note: The preceding command adds 10 new Pods and schedules them on the new CIDR range on new worker nodes.

Important: It can take a cluster up to 5 hours to recognize and allowlist a new CIDR block that you associate with a VPC. During that time, the Amazon EKS control plane can't communicate with the Pods that you launch in the new subnets. If api-server contacts Pods with a webhook, kubectl logs command, or kubectl exec command, then you might receive the following error message:

"failed calling webhook "linkerd-proxy-injector.linkerd.io": failed to call webhook: Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=10s": Address is not allowed"