How do I receive notification when the IP address of an Application Load Balancer changes?

Short description

To receive notifications when load balancer IP addresses change, create an Amazon EventBridge rule that monitors AWS CloudTrail for elastic network interface changes. To prevent disruptions, access your load balancer through the DNS server instead of directly through IP addresses.

Resolution

Resolve firewall and security group allowlist disruptions

Your downstream firewalls, network access control lists (network ACLs), or third-party security appliances might use an IP address allowlist for the load balancer. When an IP address changes, the rules fail and cause connectivity failures between the load balancer and backend services.

To resolve this issue, don't allowlist IP addresses for the load balancer. If the firewall is within the same virtual private cloud (VPC), then configure your firewall rules to allow traffic from the security group of the load balancer.

For a firewall that's external to the VPC, identify the CIDR block of the subnet where you deploy the load balancer. Then, add the CIDR block to your firewall's allowlist.

Resolve stale DNS responses from cached IP addresses

It might take your application, DNS resolver, or client longer than the 60-second time-to-live (TTL) to cache the DNS response from the load balancer. Traffic then continues to route to inactive IP addresses and causes connection timeouts, refused connections, or intermittent connectivity failures.

To resolve this issue, take the following actions:

• Clear the DNS cache on the client operating system (OS).
• Configure HTTP clients, connection pools, and frameworks to use a maximum cache duration of 60 seconds.
• Confirm that third-party DNS resolvers use the 60-second TTL from the load balancer.
  Note: Some HTTP clients, connection pools, and frameworks such as Java, .NET, or NGINX independently cache DNS responses. Also, some ISP or corporate DNS resolvers override TTL values.

Resolve direct IP address references in your configuration

If your application, load balancer configuration, or infrastructure-as-code directly references IP addresses instead of the DNS name, then connectivity fails when IP addresses change.

You must use the DNS name of the load balancer. To use a friendly DNS name, create a CNAME record or an Amazon Route 53 alias record that points to the DNS name of the load balancer.

Resolve health check failures after IP address changes

Update the health check settings of the target security group to allow traffic from the security group of the load balancer, instead of specific IP addresses.

The load balancer sends health checks from its assigned IP addresses. When the IP addresses change, your targets must accept traffic from the new IP addresses. If your target security groups or application-level firewalls restrict health check traffic by source IP address, then health checks fail. The load balancer then marks the targets as unhealthy.

Resolve intermittent connectivity during scaling events

During a scaling event or IP address change, some clients might still connect to inactive IP addresses when DNS propagates. Intermittent connectivity failures, slow response times, and reduced transaction success rates can occur.

To resolve this issue, confirm that your clients have retry logic with exponential backoff and jitter. Configure connection pools to periodically refresh the DNS. If you use a proxy to forward traffic to the load balancer, such as NGINX or HAProxy, then configure the proxy to dynamically resolve the DNS. Don't configure the proxy to resolve the DNS at startup.

Use best practices for static IP addresses

If your architecture requires static IP addresses, then take one of the following actions:

• Place a Network Load Balancer with Elastic IP addresses between your clients and the Application Load Balancer.
• Use AWS Global Accelerator to provide two static anycast IPv4 addresses.
• Use Application Load Balancer integration with Amazon VPC IP Address Manager (IPAM) to assign IP addresses from a known CIDR block.

Network Load Balancer with the Application Load Balancer

Create a Network Load Balancer, and assign its Elastic IP addresses to each Availability Zone. Then, create an Application Load Balancer type target group on the Network Load Balancer, and register your Application Load Balancer as the target.

The Network Load Balancer provides static IP addresses and the Application Load Balancer handles layer 7 routing, host-based routing, and path-based routing. When clients and firewalls reference the static IP addresses of the Network Load Balancer, changes to IP addresses of the Application Load Balancer don't affect connectivity.

Global Accelerator

If you require static IP addresses to serve global traffic, then use Global Accelerator to create an accelerator with your Application Load Balancer as an endpoint. Global Accelerator provides two static anycast IPv4 addresses that don't change. Traffic routes over the AWS global network to improve latency and availability.

Application Load Balancer integration with Amazon VPC IPAM

The Application Load Balancer gets public IP addresses from an IPAM pool with a known CIDR block, such as BYOIP /28. You can provide the CIDR block to external partners and customers to add to their allowlist. The IP addresses of the load balancer always comes from the CIDR block.

Use bring your own IP addresses (BYOIPs) with the load balancer. When you use BYOIP, you don't incur the hourly charge for Amazon provided public IPv4 addresses. If you don't use BYOIP, then you can use Amazon provided contiguous prefixes in an IPAM pool to get a predictable and narrow IP address range.

When the load balancer scales and adds new nodes, the new nodes gets additional IP addresses from the same IPAM pool. If the pool is exhausted, then the load balancer returns to AWS Regional IP addresses.

Set up notifications for IP address changes

When IP addresses change, AWS creates or deletes network interfaces in the subnets of the Application Load Balancer. CloudTrail logs the actions as CreateNetworkInterface and DeleteNetworkInterface events. To match the events and send notifications through Amazon Simple Notification Service (Amazon SNS), create an EventBridge rule.

Prerequisite: Create a CloudTrail trail in the AWS Region where you deployed your load balancer. The trail must log management events for EventBridge to receive the API call events. Also confirm that you created an Amazon SNS topic with a confirmed email subscription.

Create an EventBridge rule that matches network interface events where the description starts with ELB app/.

Note: The ELB app/ prefix identifies network interfaces that the load balancer manages.

When you configure the event pattern for the rule, use the following JSON:

aws events put-rule \
    --name alb-eni-change-rule \
    --event-pattern '{
        "source": ["aws.ec2"],
        "detail-type": ["AWS API Call via CloudTrail"],
        "detail": {
            "eventSource": ["ec2.amazonaws.com"],
            "eventName": ["CreateNetworkInterface", "DeleteNetworkInterface"],
            "requestParameters": {
                "description": [{"prefix": "ELB app/"}]
            }
        }
    }' \
    --state ENABLED \
    --region REGION

Note: Replace REGION with your Region. The pattern filters for network interface creation and deletion events that are specific to the load balancer.

Run the following command to select your SNS topic as the target for the rule:

aws events put-targets \
    --rule alb-eni-change-rule \
    --targets "Id"="1","Arn"="arn:aws:sns:us-east-1:ACCOUNT_ID:alb-ip-change-notifications" \
    --region REGION

Note: Replace REGION with your Region and ACCOUNT_ID with the AWS account ID where you deployed your load balancer.

Then, add the following resource-based policy to the SNS topic to grant EventBridge permission to publish to the topic:

aws sns set-topic-attributes \
    --topic-arn arn:aws:sns:us-east-1:ACCOUNT_ID:alb-ip-change-notifications \
    --attribute-name Policy \
    --attribute-value '{
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "AllowEventBridgePublish",
            "Effect": "Allow",
            "Principal": {"Service": "events.amazonaws.com"},
            "Action": "sns:Publish",
            "Resource": "arn:aws:sns:us-east-1:ACCOUNT_ID:alb-ip-change-notifications"
        }]
    }' \
    --region REGION

After you create the rule, you receive an SNS notification when AWS creates or deletes a network interface for your load balancer. The notification includes event details such as the network interface ID, subnet, and description that identifies the affected Application Load Balancer.

Related information

How Elastic Load Balancing works

Use an Application Load Balancer as a target of a Network Load Balancer

Security group referencing

Simplify ALB's public IP address assignment with VPC IPAM