I have questions on Kubernetes ENI allocation

0

Via https://github.com/aws/amazon-vpc-cni-k8s, I found how ENI allocation based on instance type. It described it by using m4.4xlarge as an example.

For example, a m4.4xlarge node can have up to 8 ENIs, and each ENI can have up to 30 IP addresses. ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html ).

If the number of current running Pods is between 0 to 29, ipamD will allocate one more eni. And Warm-Pool size is 2 eni * (30 -1) = 58

If the number of current running Pods is between 30 and 58, ipamD will allocate 2 more eni. And Warm-Pool size is 3 eni * (30 -1) = 87

I did calculate it further below. Please correct me if I am wrong.

Q1) m4.4xlarge will have maximum pods will be 203 and Warm-pool will have 232. Is that right?

Q2) If the pods will be over 203, what will be happened? Does new instance(m4.4xlarge) will be launching?

Q3) Why this calculation get started from '0(zero)' pod? please

PodsWarm-PoolCalculation
0 - 29582eni * (30 - 1)
30 - 58873eni * (30 - 1)
...
146 - 1742037eni*(30 - 1)
175 - 2032328eni * (30 - 1)

Thank you

asked 5 years ago1008 views
1 Answer
0
Accepted Answer

It is now possible to have greater control over how many ENI's are grabbed by the CNI.

On the other hand, Kubernetes itself has a supported limit of 100 pods per node. Making some of the larger instances with lots of available addresses less attractive. While the pod per node limit IS configurable, I would not increase it without a very good reason. This implies that the best instances sizes are between 2xlarge-4xlarge, at least in terms of address allocation. Larger sizes may be better in terms performance, but you will not get any more useful addresses.

https://kubernetes.io/docs/setup/cluster-large/ No more than 100 pods per node

Q1) a m4.4xlarge node can have up to 8 ENIs, and each ENI can have up to 30 IP addresses. ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html ). This is 240 addresses maximum for pods. The warm pool is controlled by "WARM_ENI_TARGET", which is defaults to 1. This will imply 30 addresses in the warm pool (the number available to an ENI). However, at maximum addresses, there will be no warm pool left. The warm pool target is configurable by en ENV variable, and the algorithm has been tweaked recently. I would test this out to verify the actual numbers if there is concern.

Q2) Nothing is done automatically. You can run out of addresses in a K8 cluster. You may want to look into "cluster autoscaling" which will launch new hosts upon address exhaustion (or pod placement failure to be more accurate).

Q3) One address allocated to the ENI is considered the Primary, and it used for routing traffic out of the worker node. This is why you lose an address per ENI.

MODERATOR
answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions