EKS VPC-CNI Plugin Node Group Setup Questions

0

I am creating an EKS managed node group in terraform using the eks module version 17.1.0 and up until now specifying the bootstrap_extra_args like so has been working

    node_groups = [{
            
            name = "${var.environment}-nodes"
            desired_capacity = var.eks_cluster.desired_capacity
            max_capacity     = var.eks_cluster.max_capacity
            min_capacity     = var.eks_cluster.min_capacity
        
            additional_security_group_ids = aws_security_group.nodes.id
            instance_types                = [var.eks_cluster.node_instance_type]
            key_name = "$$$$$$"
            bootstrap_extra_args = "/etc/eks/bootstrap.sh '${local.cluster_name}' --use-max-pods false --kubelet-extra-args '--max-pods=110'"
    }]

I have created two clusters like this and the nodes have been created with the max pods set to 110. both of these clusters are in us-east-2

I am now trying to create a cluster in China region cn-northwest-1 and the same configuration only sets the max pods to 35 and I cannot seem to get it to go any higher.

Node types: t3a.large instances

Note: I have also attempted to launch the nodes in China with a launch_template containing the following userdata script and the script is read, there are no errors that I can find and I end up with the same result.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash -xe
/etc/eks/bootstrap.sh '${cluster_name}' --use-max-pods false --kubelet-extra-args '--max-pods=110'
--//--

This begs the question, are eks managed node groups setup a bit differently in china? Is what I'm trying to do even possible without some crazy workaround I cannot seem to find?

asked 2 years ago863 views
1 Answer
0

Hello,

Depending on which instance family and size you run, you have a constraint of number of IPs per ENI and a max number of ENI applies too.

As described here : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

For example, a t3a.large instance is constrained to 3 ENI and 12 IPs per ENI at max.

Do you use the same instance family/size for your clusters in us-east-2 ?

For ref, a great blog post that explains the mechanisms of CNI IP allocation and how to manage the limits with prefixes.

https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/

Thanks

AWS
answered 2 years ago
  • yup I used the same family/size (t3a.large) in us-east-2. Hence my confusion that its behaving differently, but we have also noticed that the console in us-east-2 mentions prefixes for each network interface whereas the console in cn-northwest-1 does not. This makes me think that the prefix assignment feature is not available in the cn-northwest-1 region which could be the issue here possibly?

  • Hi, i don't see any region specific limitation for prefix assignment on my end.

    If you already use this capability without any issue, then i would take a look at VPC CNI logs just to make sure the plugin can operate as expected for allocation.

    ie: troubleshooting scenarios described here : https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions