TGW Appliance Mode with GWLB

Question

The default behavior of AWS TGW that it will try to maintain the Availability zone when forwarding traffic so also know as : Zone preservation.

Traffic comes from attachment in Zone1 then it sends to TGW ENI in the same AZ when sending it to the target attachment.

But with Appliance mode enabled :
When appliance mode is enabled, a transit gateway selects a single network interface in the appliance VPC, using a flow hash algorithm, to send traffic to for the life of the flow. The transit gateway uses the same network interface for the return traffic.

When appliance mode is not enabled, a transit gateway attempts to keep traffic routed between VPC attachments in the originating Availability Zone until it reaches its destination.

What I understood is that if appliance mode is enabled then TGW will try to keep the traffic in a single AZ for the lifetime of the session.  In the case of appliance VPC  for example TGW will use a hash algorithm to pick the NIC inside the shared VPC attachment.

My question is related to GWLB :
When we have GWLB - the GWLB is supposed to keep the traffic balanced to the same target instance as long as it is the same flow and load balancer. Let us say I do not have appliance mode enabled and then :
1- Traffic comes from source in VPC1 in AZ1 to TGW 
2- TGW forwards it to TGW attachment NIC in AZ1 .
3- From that attachment then it is passed to the GWLB endpoint then to GWLB and then to VM1.
4- Assuming traffic arrives to destination in AZ2 in VPC 2. return traffic is routed back to TGW attachment in VPC2 using AZ2 .
5- Transit gateway preserves the AZ and send it to TGW attachment NIC in AZ2. then to GWLB endpoint in AZ2 and then to GWLB.

Should not the GWLB forward it back to to the same VM serving the request ? or that session is not preserved if being served by different GWLB endpoints ?

NeedleStack · Answer

NLB has a worker node in each AZ . So if you have NLB with 2 AZs and it is an internet facing one. When you resolve you get two IPs . This scenario below:
1- session 1 resolves to IP1 and goes to AZ1
2- session 2 resolves to IP2 and goes to AZ2 
stickiness is enabled and cross zone load balancing too.

Does this mean both sessions will be sent to the same backend?

Brettski-AWS · Answer

The problem that Transit Gateway Appliance Mode solves is ensuring that traffic flows to and from Gateway Load Balancer are symmetrical.

If you have two instances (A and B) in different AZs then Transit Gateway may well select a different path for traffic from A to B as compared to B to A. This could mean that traffic from A to B goes to Gateway Load Balancer Endpoint X; but traffic from B to A goes to Gateway Load Balancer Endpoint Y. And in that case they may (quite probably) be sent through different firewalls.

Firewalls are stateful. They really don't like it when traffic from A to B is seen but traffic from B to A is not. And vice versa. You'll have a situation where something stateless like ICMP (and some UDP traffic) works fine; but TCP traffic does not. The first firewall will see the TCP SYN from A to B but not see the SYN-ACK back from B. The second firewall will probably drop the SYN-ACK becuase it didn't see the SYN first. All sorts of things will go wrong.

Enabling Appliance Mode ensures that traffic flows from A to B and from B to A are routed to a Gateway Load Balancer Endpoint in the same VPC regardless of which Availability Zones instances A and B are in.

TGW Appliance Mode with GWLB

Relevant content