Clarification on Security Groups for pods and Karpenter Behavior How maxPods is calculated when pod_eni is enabled?

Question

Hello,

I’ve read through AWS documentation [https://docs.aws.amazon.com/eks/latest/best-practices/sgpp.html], and I’m trying to reconcile it with Karpenter’s explanation regarding ENIs and pod density.

The documentation mentions that “branch interface capacity is additive to existing instance type limits for secondary IP addresses.” However, according to Karpenter’s documentation, the behavior seems to be the opposite. For example, Karpenter notes the following:

Note: If you’ve enabled Security Groups per Pod, one of the instance’s ENIs is reserved. To avoid discrepancies between the maxPods value and the node’s supported pod density, you need to set RESERVED_ENIS=1.

Additionally, Karpenter provides further clarification:

If you’ve enabled Security Groups per Pod, one of the instance’s ENIs is reserved as the trunk interface, which uses branch interfaces to assign different security groups. If no SecurityGroupPolicies are configured, pods cannot use the branch interfaces attached to the trunk ENI, and IPs are only available from the non-trunk ENIs.

This effectively reduces the maximum pod capacity compared to what would normally be supported by the instance type. Karpenter is also not aware when Security Groups per Pod is enabled, so it continues to compute maxPods assuming all ENIs can be utilized.

Given this, there seems to be a contradiction between AWS’s statement that “branch interface capacity is additive” and Karpenter’s documentation, which suggests that enabling Security Groups per Pod reduces available IP capacity (since one ENI becomes reserved).

Can somebody clarify this behavior? Specifically:

When Security Groups per Pod is enabled, does the trunk ENI capacity truly add to the total available IPs, or does it reduce usable capacity as Karpenter suggests?

Should we always set RESERVED_ENIS=1 when using Security Groups per Pod to ensure correct pod density calculation?

I’m confused about how this applies in practice.
According to one source, an m5.large instance supports 3 ENIs, but your documentation seems to indicate that the instance could support 4 ENIs because branch interfaces are additive.

Could you please clarify which source is accurate, and how branch interfaces impact ENI limits and IP allocation?

Thank you for your help in clarifying this discrepancy.

Sources:
https://docs.aws.amazon.com/eks/latest/best-practices/sgpp.html
https://karpenter.sh/docs/concepts/scheduling/
https://karpenter.sh/docs/troubleshooting/#maxpods-is-greater-than-the-nodes-supported-pod-density
https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/v1.5.0/pkg/aws/vpc/limits.go

Best regards,
Issa Al Msirdi

re:Post Agent · Answer

When Security Groups for Pods is enabled in Amazon EKS, there are some important distinctions to understand about how ENIs and IP addresses are allocated.

The statement that "branch interface capacity is additive to existing instance type limits for secondary IP addresses" means that the branch ENIs created for Security Groups for Pods provide additional IP addresses beyond what would normally be available through regular secondary ENIs. This is because the trunk ENI (created when you enable Security Groups for Pods) and its associated branch interfaces operate separately from the regular ENI allocation.

However, Karpenter's documentation is also correct. When Security Groups for Pods is enabled, one of the instance's regular ENIs is designated as a trunk ENI. This trunk ENI is then used to create branch interfaces for Pods that need specific security groups. The trunk ENI itself is not available for regular Pod IP assignment, which is why Karpenter recommends setting `RESERVED_ENIS=1`.

To clarify:

1. When Security Groups for Pods is enabled, one ENI becomes a trunk ENI and is no longer available for regular Pod IP assignment.
2. The branch interfaces attached to this trunk ENI provide additional IP capacity that is separate from the regular ENI IP allocation.
3. Pods that use security groups are not counted in the standard max-pods calculation.

For proper pod density calculation with Karpenter, you should set `aws.reservedENIs=1` in the karpenter-global-settings configmap when using Security Groups for Pods. This ensures Karpenter correctly accounts for the ENI that will be used as the trunk ENI.

Regarding your specific example of an m5.large instance: The instance still supports 3 ENIs as per its EC2 specifications. When Security Groups for Pods is enabled, one of these becomes a trunk ENI. The branch interfaces attached to this trunk ENI provide additional capacity, but they don't increase the number of primary ENIs the instance can have.

If you're using Security Groups for Pods, you may need to consider raising the max-pods value to account for the additional capacity provided by branch interfaces, or be prepared to run fewer regular pods than the node might otherwise support.  
**Sources**  
[Security Groups Per Pod - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/sgpp.html)  
[Configure the Amazon VPC CNI plugin for Kubernetes for security groups for Amazon EKS Pods - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-pods-deployment.html)  
[Troubleshooting Amazon EKS networking issues at scale in an Enterprise scenario | AWS re:Post](https://repost.aws/articles/ARMvpWa7AYScyBBPRr6xkvXw/troubleshooting-amazon-eks-networking-issues-at-scale-in-an-enterprise-scenario)  
[Network security - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/network-security.html)  
[Amazon VPC CNI - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/vpc-cni.html)

Clarification on Security Groups for pods and Karpenter Behavior How maxPods is calculated when pod_eni is enabled?

Add your answer

Relevant content