Intermittent session drops through a stateful firewall after enabling a second Local Gateway BGP peer on AWS Outposts

Short description

You run AWS Outposts behind a stateful firewall and enable both Local Gateway (LGW) BGP peers for redundancy. Long-lived sessions (RDP, SMB, persistent HTTP, DNS over TCP) begin failing intermittently. With only one LGW peer active, everything works. Routing tables look stable, with no BGP flapping and no link errors.

The cause is an asymmetric return path. When both LGW peers are up and your firewall advertises your on-premises prefixes to both with an equal AS-path length, the Outpost sees two equal-cost return paths and performs per-flow ECMP across them. Your firewall sends a session out one LGW sub-interface and registers the session on that interface, but the Outpost returns some flows on the other LGW. Those return packets arrive on an interface the session table does not expect, fail the stateful inspection check, and are dropped.

Disabling the second peer restores symmetry and appears to fix the issue, which misleads operators into treating the second peer as faulty. The peer is fine. The asymmetry is the problem.

The following diagram shows both states side by side: the asymmetric flow that causes the intermittent drops, and the symmetric flow that results once the fix is applied.

Figure 1: Return-path asymmetry through a stateful firewall with dual Local Gateways, and the AS-path prepend that restores a single symmetric path.

Resolution

Work through the following steps in order: confirm the asymmetry, apply the prepend on the backup peer, verify the advertised routes, and validate failover before you call it done.

Step 1: Confirm the asymmetry

Two checks separate this from the causes it is often mistaken for (flapping, MTU, multipath tuning).

Rule out route flapping. Compare the best path and its uptime for a VM prefix in the working (one LGW) and failing (two LGW) states. If the best path and uptime are unchanged when the second peer comes up, the prefixes are not oscillating and flapping is not the cause.

Confirm the split directly. Capture on both LGW sub-interfaces at once, filtered to one failing session's 5-tuple. On a Cisco device:

capture asym1 interface <LGW1_SUBIF> match tcp host <client-ip> host <vm-ip>
capture asym2 interface <LGW2_SUBIF> match tcp host <client-ip> host <vm-ip>
show capture asym1
show capture asym2

If the SYN leaves on one LGW and the SYN-ACK returns on the other, the asymmetry is confirmed.

Step 2: Apply an outbound AS-path prepend on the backup LGW peer

Make the Outpost prefer one LGW for all return traffic by advertising your on-premises prefixes with a longer AS-path on the backup peer. Three prepends is the common convention.

Cisco IOS / FTD:

route-map LGW1-PREPEND-OUT permit 10
 set as-path prepend <own-ASN> <own-ASN> <own-ASN>
!
router bgp <own-ASN>
 address-family ipv4 unicast
  neighbor <LGW1-peer-IP> route-map LGW1-PREPEND-OUT out

FortiOS:

config router route-map
    edit "LGW1-PREPEND-OUT"
        config rule
            edit 1
                set set-aspath "<own-ASN>" "<own-ASN>" "<own-ASN>"
            next
        end
    next
end
config router bgp
    config neighbor
        edit "<LGW1-peer-IP>"
            set route-map-out "LGW1-PREPEND-OUT"
        next
    end
end

Then soft-clear the session outbound so the new AS-path is advertised immediately (clear ip bgp <LGW1-peer-IP> soft out on IOS, execute router clear bgp ip <LGW1-peer-IP> soft out on FortiOS).

Important: if your firewall already applies an outbound route-map to the LGW peers (for prefix filtering, communities, or other attributes), do not replace it. Clone it, add the prepend to the clone, and apply the clone to the backup peer only. Replacing an existing outbound policy is the most common way a prepend accidentally withdraws prefixes and takes the link down.

Step 3: Verify

• Both BGP sessions remain established.
• The advertised-routes list to the backup peer shows your prefixes with the extended AS-path, and contains the same prefixes as before the change. If any prefix is missing versus the pre-change baseline, roll back.

show bgp neighbors <LGW1-peer-IP> advertised-routes

• All Outpost VM return routes resolve to the primary LGW next-hop.
• The intermittent drops stop.

Step 4: Validate failover

Shut down the primary LGW peer. Confirm the Outpost converges onto the backup LGW and traffic recovers. New sessions use the backup immediately; existing sessions re-establish after convergence. Bring the primary back and confirm preference returns.

Rollback

Remove the route-map from the neighbor (or restore the original), then soft-clear. The change is reversible in seconds, which makes it safe to apply in a short maintenance window.

Deploying at multiple sites

The configuration is identical at any site with the same topology (stateful firewall plus dual LGW). Only the local peer IP and ASN change.

Alternatives

• MED achieves the same result with a cleaner failure mode (a misconfigured MED tends to have no effect rather than withdrawing prefixes) when both LGW peers are in the same ASN.
• A more-specific prefix on the preferred LGW pins return traffic by longest match.

A note on firewall-side "fixes"

Enabling asymmetric routing on the firewall (set asymroute enable on FortiGate, asymmetric-path bypass on Palo Alto, TCP state bypass on FTD) also stops the drops, but it does so by relaxing stateful inspection. In regulated environments this is typically prohibited, and correctly so. The BGP-side fix preserves full inspection and is the recommended approach.

Related information

• AWS Outposts High Availability Design and Architecture Considerations (see the Networking section)
• AWS Outposts rack network requirements (covers the service link and firewall rules)