How do I troubleshoot BGP connection failure over Site-to-Site VPN or Direct Connect?

Short description

To troubleshoot errors related to Border Gateway Protocol (BGP) when you configure dynamic AWS Site-to-Site VPN and Direct Connect connections, check your BGP logs. Then use your findings to identify the point of failure in the BGP negotiation process.

Resolution

Prerequisite: Complete the steps in the To enable tunnel BGP logging on an existing Site-to-Site VPN connection section of Enable AWS Site-to-Site VPN logs. Be sure to activate Site-to-Site VPN logging and BGP logging before you troubleshoot your connection.

Troubleshoot failed or stuck BGP connections over VPN tunnels and over Direct Connect.

Troubleshoot your errors with BGP logs

Check your BGP logs and use your findings to troubleshoot errors with Site-to-Site VPN tunnels.

Check your BGP configuration

On your customer gateway, confirm that you use a verified BGP configuration, and that the pings between the BGP peer IP addresses function correctly. Collect packet captures for traffic between the BGP peer IPs from your customer gateway device. Then, analyze the data for each BGP state.

Troubleshoot BGP states

Troubleshoot the Idle state

In the Idle state, BGP waits for a start event. Start events include when you configure a new BGP neighbor or when you reset peering for an established BGP. After a start event, BGP initializes resources, resets a ConnectRetry timer and initiates a TCP connection to the remote BGP neighbor. After BGP successfully initiates the TCP connection, The BGP state changes to Connect.

Troubleshoot the Connect state

In the Connect state, BGP waits for the TCP three-way handshake to complete. When the connect state successfully completes, the connection moves to the OpenSent state. The following example packet capture shows that the TCP three-way handshake successfully completes:

2021-07-04 22:50:20 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081

If the connection or ConnectRetry fails, then BGP remains in the Active state and doesn't go to OpenSent state.

If the connection fails, then check the logs for your customer gateway device for a TCP handshake. Check your Site-to-site VPN logs and BGP logs, and then take the following troubleshooting actions based on your findings:

• For dynamic VPN, perform a telnet test on TCP port 179 to check TCP connectivity between the two BGP neighbors. If there isn't TCP connectivity, then check the logs for errors or packets that drop during the TCP connection.
• Check that you correctly configured the neighbor's IP address on BGP, the customer gateway, and AWS.
• If you use the Direct Connect virtual interface, then check if you entered the correct BGP authentication, or MD5 password, on the routers.

Troubleshoot the OpenSent state

After BGP sends an OPEN message to the peer, BGP waits in the OpenSent state for the OPEN reply. If BGP receives a successful reply, then the BGP state moves to OpenConfirm and sends a KEEPALIVE message to the peer. If the connection fails, then BGP reverts to the Idle or Active state.

In the OpenSent message, the peer sends the following BGP parameters:

1. version number
2. AS number
3. Hold Time
   Note: The default Hold Timer value is 30 seconds for BGP over VPN and 90 seconds for BGP over Direct Connect. AWS doesn't accept 0 as a Hold Time value.
4. BGP Identifier IP address

See the following example OpenSent message:

Border Gateway Protocol - OPEN Message
Type: OPEN Message (1)  
Version: 4   
My AS: 65000  
Hold Time: 30  
BGP Identifier: 54.241.242.80

If BGP doesn't establish a connection, then in your BGP logs, check that BGP correctly sent the OPEN message. Also, check that the neighbor correctly received the OPEN message that contains the BGP parameters. Check the OpenSent logs to identify the cause of failure.

Troubleshoot the OpenConfirm state

BGP waits in the OpenConfirm state for KEEPALIVE messages from the peer. If BGP receives a KEEPALIVE message, then the state moves to the final Established state. If BGP doesn't receive a KEEPALIVE message the state moves back to Idle or Active state, based on the errors.

The following example packet capture shows that two BGP peers successfully exchanged KEEPALIVE messages, and the status of BGP can move from OpenConfirm to Established:

65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message

In the logs, check that BGP correctly sent and received the KEEPALIVE message.

Troubleshoot the Established state

In the Established state, BGP exchanges information between the peers, such as updates, or route advertising, KEEPALIVES, and notifications.

The following example packet capture shows that the BGP established a session correctly, and the BGP peers successfully exchanged messages:

Border Gateway Protocol - UPDATE Message
Path attributes
Path Attribute - AS_PATH: 65000 
Path Attribute - NEXT_HOP: 169.254.60.146 
Network Layer Reachability Information (NLRI)
   192.168.0.0/16 

If a connection doesn't establish, then complete the following checks:

• In the logs, check that routers correctly exchange updates, and that the advertised prefixes match the expected routes.
• Check that no BGP filters or prefix lists prevent routes from propagating in the route table.
• Check the advertised route entries on peer route tables.
• For a Site-to-Site VPN or Direct Connect Virtual Interface on a virtual gateway, check the state of the BGP session. If the state changes from Established to Idle, then check that the peer advertises fewer than 100 routes over the session.