Site-to-Site VPN Convergence

Hello,

We currently use a redundant site-to-site VPN with BGP between between AWS and our on-prem site. Each site-to-site VPN from AWS side connects to a single WAN endpoint on our side. We have configured BGP with LP and MED to make sure the flow is symmetric with the preference as shown in the image below. BGP is configured with a keep-alive of 3 and hold-down timer of 9 seconds while DPD is configured with 10 seconds interval and retry count of 3.

While doing failover testing I observed that whenever WAN1 goes down, it takes approximately 20-35 seconds for the AWS side to converge. I can see our side firewall using the new tunnel and BGP route as soon as 10 seconds. Further decreasing the BGP timers and the IPSEC DPD timers don't bring much change to this time. I have a trace/mtr running from inside the VPC and I can see that AWS side keeps using primary tunnel even when the tunnel is down on AWS side. The TGW route-table does update prompty with the new tunnel attachment but the traceroute keeps using the old down tunnel. I have tried MTR with different options like TCP/ICMP/UDP to make sure it's not some weird ICMP hashing issue.