Urgent Assistance Needed – Tunnel Down Since Monday - IPSEC IS DOWN

To troubleshoot your down IPsec tunnel, I recommend checking several key areas:

1. **DPD (Dead Peer Detection) Timeout:

• Please verify if your customer gateway device is responding to IKE keep alive/DPD messages. If the device does not respond within the DPD timeout interval, AWS marks the tunnel as down.

2. **Peer Device Responsiveness:

• Check CGW On-prem Device VPN debug logs to understand the Tunnel Down reason.
• Check the logs on your customer gateway device to confirm whether it is receiving and responding to IKE negotiations from AWS.
• Ensure that UDP ports 500 (IKE) and 4500 (IPsec) are open and not blocked by any firewall or security policy.

3. **Enable CloudWatch logging:

• Activate CloudWatch logging for your VPN connection on AWS side to check for issues like Dead Peer Detection (DPD) timeouts or other clear indicators of what might be causing the tunnel to fail.

4. **Check Personal Health dashboard:

• Your Site-to-Site VPN connection consists of two VPN tunnels for redundancy. Sometimes, one or both of the VPN tunnel endpoints is replaced when AWS performs tunnel updates, or when you modify your VPN connection. During a tunnel endpoint replacement, connectivity over the tunnel might be interrupted while the new tunnel endpoint is provisioned.
• Check if tunnel got replaced under maintenance and post that it remained Down.

Since no configuration changes were made, it's possible that something in the network path between your gateway and AWS has changed, or there might be an issue with the customer gateway device itself.

5. **Network Path Reachability:

• Run a reachability test (ping/traceroute) from your on-premises device towards the AWS VPN IPs to confirm if there is packet loss or network path issues.

Additionally, I would also recommend opening a support case with AWS Support so that the VPN connection can be reviewed in detail.

Sources
IP Sec site to site Tunnel instability | AWS re:Post
AWS Site to Site VPN is not up and running | AWS re:Post
VPN stopped working - not receiving ESP traffic back from AWS | AWS re:Post
Troubleshoot AWS Site-to-Site VPN connectivity with a Cisco IOS customer gateway device - AWS Site-to-Site VPN

AWS Site-to-Site VPN tunnel endpoint replacements

To troubleshoot a down VPN tunnel in AWS, you can use several AWS CLI commands to gather information and diagnose the issue. Here are some helpful commands:

Describe the VPN connection:

#aws ec2 describe-vpn-connections --vpn-connection-ids <vpn-connection-id>

This will give you an overview of the VPN connection, including its current state and configuration.

Describe the Customer Gateway:

#aws ec2 describe-customer-gateways --customer-gateway-ids <customer-gateway-id>

This provides information about the customer gateway associated with the VPN.

Describe the Virtual Private Gateway:

#aws ec2 describe-vpn-gateways --vpn-gateway-ids <vpn-gateway-id>

This gives details about the VPN gateway on the AWS side.

Check VPN tunnel logs:

#aws ec2 get-vpn-connection-device-types #aws ec2 get-vpn-connection-device-sample-configuration --vpn-connection-id <vpn-connection-id> --vpn-connection-device-type-id <device-type-id>

These commands can help you get sample configurations and check for any misconfigurations.

Describe route tables:

#aws ec2 describe-route-tables --filters Name=route.state,Values=active Name=route.origin,Values=CreateRoute

This helps verify if the routes for the VPN are properly set up in your VPC.

Check Network ACLs:

#aws ec2 describe-network-acls --filters Name=vpc-id,Values=<vpc-id>

Ensure that the Network ACLs are not blocking the traffic.

Describe security groups:

#aws ec2 describe-security-groups --group-ids <security-group-id>

Verify that the security groups are allowing the necessary traffic.

Check CloudWatch metrics for the VPN connection:

#aws cloudwatch get-metric-statistics --namespace AWS/VPN --metric-name TunnelState --dimensions Name=VpnId,Value=<vpn-connection-id> Name=TunnelIpAddress,Value=<tunnel-ip> --start-time <start-time> --end-time <end-time> --period 300 --statistics Average

This can show you the state of the tunnel over time.

Describe VPN connection route:

#aws ec2 describe-vpn-connections --filters Name=vpn-connection-id,Values=<vpn-connection-id> --query 'VpnConnections[].Routes[]'

This shows the routes advertised by the customer gateway.

Remember to replace placeholder values (in <>) with your actual resource IDs.

After running these commands, analyze the output for any discrepancies, error messages, or unexpected configurations. Pay special attention to:

• The state of the VPN connection and tunnels
• Any changes in IP addresses or routing information
• Security group and NACL rules that might be blocking traffic
• Metrics indicating when the tunnel went down

If you're still unable to identify the issue, you may need to:

Check your on-premises equipment for any changes or issues Review AWS Health Dashboard for any relevant events Contact AWS support for further assistance, especially if you suspect an issue on the AWS side