Announcing Support for Multiple Secondary IP Addresses for Network Load Balancer Egress Interface

Introduction:

Today, we are announcing the capability to use multiple secondary IPv4 addresses on Network Load Balancers (NLB). This capability allows the configuration of up to 7 additional IPv4 addresses on the egress interface of NLB, which increases NLB's concurrent connections per target from 64K to 512K, and reduces the probability of TCP ephemeral port allocation errors, for scenarios where Source IP preservation cannot be used.

Feature deep-dive:

Customer Challenge:

When client source IP preservation is disabled, NLB uses its own IP address and ports from the TCP ephemeral port range to route connections to backend targets, similar to a NAT Gateway. In IPv4, the TCP ephemeral port space is 2 bytes, which restricts the NLB to create up to approximately 64K simultaneous connections to each target.

When the number of source ports to each target IP reaches the maximum, the NLB starts to reject new flows as it cannot allocate a new port, and increments the PortAllocationErrorCount metric. Prior to this feature, the only solution to fix port allocation errors was to add more targets to the target group.

This scenario is always present in environments that use PrivateLink, as connections originating from PrivateLink Endpoints do not preserve the client Source IP.

Some workloads need more connections and customers had asked us to solve this IPv4 TCP port limitation. See Figure 1 below that shows today’s NLB behavior with single egress IP address.

Solution:

To solve this limitation, using the new capability NLB can now allocate to up to 8 total IPs (1 default + 7 secondary) when communicating with targets. The use of additional IPv4 addresses allows customers to overcome the 64K port constraint, and increases the number of concurrent connections from 64K to 512K per target. See Figure 2 below that shows NLB behavior with this new capability with multiple secondary IP address.

Summary of feature benefits:

• Increased connection capacity: Supports up to 8 times more concurrent connections per target
• Improved scalability: Reduce the need to add more targets solely for port allocation purposes
• Flexibility: Ability to gradually add more IP addresses as your traffic grows

Considerations and Best Practices:

Key considerations when deploying this feature:

1. This feature allocates additional IP addresses for communication with targets; the NLB DNS is not updated to serve traffic from these additional IPs, and no new Public IPs are allocated.
2. All NLB AZs are updated at once, you cannot select specific subnets where you want to enable this feature.
3. The secondary IPs are randomly selected from the NLB subnet, and you cannot choose the specific IP addresses to be used.
4. Health checks are randomly sent through any of the IPs that can route traffic to your target group.
5. You can enable from 1 to up to 7 secondary addresses, and this can be done gradually based on your traffic growth pattern.
6. The feature does not support allocating additional IPv6 addresses for NLB.
7. This feature applies to TCP and TLS listeners, and not to UDP as UDP requires the use of Source IP Preservation.
8. Once enabled, this feature cannot be disabled.

We recommend adhering to the following best practices:

1. Thoroughly test the feature in a non-production environment before deploying it in a production setting.
2. If your targets can accept traffic from multiple IP addresses, allow the subnet IP addresses of your NLB to ensure proper traffic flow. Ensure that your security groups and Network Access Control Lists (NACLs) are configured to permit traffic from all NLB Secondary IP addresses.
3. Only utilize this feature on Network Load Balancers with Source IP Preservation disabled or when using VPC PrivateLink.
4. Verify that you have sufficient available IP addresses in your NLB subnet before enabling this feature. You can use Network Address Usage for your VPC for tracking these.
5. Consider starting with a very small number of secondary IP addresses and gradually increasing them as needed.
6. Review and adjust the number of secondary IP addresses based on your traffic pattern growth rate.
7. Ensure that you have the necessary IAM permissions for ModifyLoadBalancerAttributes.

How to enable this feature?

Now that you understand the prerequisites, best practices and considerations, it's time to learn how to enable this feature. In this post, we cover AWS Console and CLI configuration to enable this feature.

Using AWS Console:

To enable this feature in the AWS Console: Locate your NLB and select Load Balancer Attributes

1. Click Edit and go to Special Case Attributes
2. Enable Secondary IP addresses auto-assigned per subnet
3. Specify the number of IPv4 addresses to use, you can start with 1 secondary IP address and gradually increase if needed.

To verify the configuration:

1. Locate your NLB and select Load Balancer Attributes
2. View the Special Case Attributes setting at the bottom of the console

Using the AWS CLI:

aws elbv2 modify-load-balancer-attributes
--load-balancer-arn arn:aws:elasticloadbalancing:us-west-2:123456789012:loadbalancer/net/my-nlb/1234567890123456
--attributes Key=secondary_ips.auto_assigned.per_subnet,Value=2

You can review the IPs by using ec2 describe-network-interfaces API to pull the list of primary and secondary IPs assigned to the NLBs ENI:

aws ec2 describe-network-interfaces --region us-east-1--filters Name=network-interface-id,Values=eni-0a1b2c3d4e5f6
{
  "NetworkInterfaces": [
    {
      "Association": {
        "IpOwnerId": "123456789012",
        "PublicDnsName": "ec2-YY-ZZZ-WWT.compute-1.amazonaws.com",
        "PublicIp": "**************"
      },
      "Attachment": {
        "AttachmentId": "ela-attach-0a1b2c3d4e5f6",
        "DeleteOnTermination": false,
        "DeviceIndex": 1,
        "InstanceOwnerId": "amazon-aws",
        "Status": "attached"
      },
      "AvailabilityZone": "us-east-1a",
      "Description": "ELB net/nlb-test1/1a2b3c4d5e6f7",
      "Groups": [],
      "InterfaceType": "network_load_balancer",
      "Ipv6Addresses": [],
      "MacAddress": "*****************",
      "NetworkInterfaceId": "eni-0a1b2c3d4e5f6",
      "OwnerId": "123456789012",
      "PrivateDnsName": "ip-172-31-23-63.ec2.internal",
      "PrivateIpAddress": "172.31.23.63",
      "PrivateIpAddresses": [
        {
          "Association": {
            "IpOwnerId": "123456789012",
            "PublicDnsName": "ec2-ZZ-YYY-WWW.compute-1.amazonaws.com",
            "PublicIp": "**************"
          },
          "Primary": true,
          "PrivateDnsName": "ip-172-31-23-63.ec2.internal",
          "PrivateIpAddress": "172.31.23.63"
        },
        {
          "Association": {
            "IpOwnerId": "123456789012",
            "PublicDnsName": "ec2-WW-YYY-ZZZ.compute-1.amazonaws.com",
            "PublicIp": "YY.ZZZ.WWW"
          },
          "Primary": false,
          "PrivateDnsName": "ip-172-31-28-219.ec2.internal",
          "PrivateIpAddress": "172.31.28.219"
        },
        {
          "Association": {
            "IpOwnerId": "123456789012",
            "PublicDnsName": "ec2-XX-WWW-ZZZ-YYY.compute-1.amazonaws.com",
            "PublicIp": "ZZ.WWW.YYY"
          },
          "Primary": false,
          "PrivateDnsName": "ip-172-31-23-202.ec2.internal",
          "PrivateIpAddress": "172.31.23.202"
        },
        {
          "Association": {
            "IpOwnerId": "123456789012",
            "PublicDnsName": "ec2-WW-YYY-ZZZ.compute-1.amazonaws.com",
            "PublicIp": "**************"
          },
          "Primary": false,
          "PrivateDnsName": "ip-172-31-31-20.ec2.internal",
          "PrivateIpAddress": "172.31.31.20"
        }
      ],
      "RequesterId": "123456789012",
      "RequesterManaged": true,
      "SourceDestCheck": false,
      "Status": "in-use",
      "SubnetId": "subnet-1a2b3c4d",
      "TagSet": [],
      "VpcId": "vpc-2b3c4d5e"
    }
  ]
}

Monitoring and Metrics:

After enabling multiple secondary IP addresses, you can monitor these CloudWatch metrics:

• PortAllocationErrorCount - If your connections had failed in the past due to PortAllocationErrors, this metric should decrease.
• ActiveFlowCount - This may increase, indicating improved connection capacity.
• NewFlowCount: This may increase, showing more successful new connections.

You can also use AWS CloudWatch alarms to alert you if PortAllocationErrorCount increases, so you can add more IPs if needed.

Region Availability

This NLB feature is available in all commercial AWS Regions, including the AWS GovCloud (US) Regions. To learn more, please refer to the NLB documentation.

Conclusion:

Enabling multiple secondary IP addresses for your Network Load Balancer enhance your application's scalability and performance. To implement this feature effectively, begin by reviewing your current NLB configuration to identify potential port allocation issues, then carefully plan your IP address strategy based on your traffic patterns. Before deploying to production, understand the caveats and thoroughly test the feature in a staging environment. When you're ready to proceed, enable the feature on your production NLB, starting conservatively with a small number of secondary IPs, and continuously monitor performance metrics to make necessary adjustments. For detailed guidance or assistance, please refer to the NLB documentation or reach out to AWS Support.