GWLB: TG Sends traffic to unhealthy instances

Question

I have the following scenario:
1 Security VPC and 1 Transit Gateway. Inside the security VPC, there are 3 AZs and 1 Gateway Load Balancer. In each AZ there is an endpoint GWLBE to redirect the traffic to the GWLB. The transit gateway attachment is configured in appliance mode. The target group of the load balancer has 3 instances: 1 VM series Palo Alto NGFW and 2 Linux machines to simulate unhealthy VM machines in each one of the AZs. The Palo Alto machine responds to the TCP 80 probes, while the other Linux machines time-out.
The target group is configured with the new failover feature to redirect existing flows to healthy instances.

Testing:
I am doing pings between 2 machines in 2 different VPCs. (I have also tested ssh traffic). I have found out that I have to send 2 to 4 times the ping command for the target machine to respond. After checking the flow logs, I found that the GWLB sends traffic to machines that are unhealthy, thus I only get the pings responses only when the GWLB send the traffic to the healthy Palo Alto.
Why is the TG sending traffic to unhealthy machines?

AWS-User-6317007 · Answer

Do you have Cross-Zone Load Balancing enabled?

--When to use Cross-Zone Load Balancing--
By default, the load balancer distributes traffic evenly across registered appliances within the same AZ. In this configuration, customers typically register more than one target within a single AZ behind the GWLB for firewall service availability and to distribute the traffic. In the event of a single target appliance failing health checks, the GWLB will route traffic to other healthy instances within the same AZ. This provides a cost-effective solution because the traffic does not cross AZ boundaries. While this setup is cost-effective, customers lose both the high availability and traffic distribution aspects in the event that all the targets in a specific AZ fail.

In order to achieve high availability and balanced traffic distribution, some customers choose another approach by enabling a feature called “cross-zone load balancing”. This feature makes it easier for you to deploy and manage your applications across multiple AZs. When you enable cross-zone load balancing, GWLB distributes traffic across all registered and healthy targets regardless of which AZs these targets are in. Enabling cross-zone load balancing incurs standard inter-AZ charges when the traffic crosses an AZ.

junjie-aws · Answer

How do I configure the Palo Alto so it forwards back the packet to where originally came from?

1) you will need to establish full mesh geneve tunnels between gwlb eni in 3 AZs and palo alto data interface
2) configure specific subnets routes for subnets in those 3 AZs, each next hop will need to be the gateway IP of that subnets

do those steps will help you forwards back the packet to where originally came from

GWLB: TG Sends traffic to unhealthy instances

Relevant content