How do I troubleshoot routing issues when my transit gateway sends traffic through inspection firewalls?

Resolution

Traffic doesn't correctly route between VPCs and your on-premises network through your transit gateway if the firewall policies are insufficient or incorrect. Also, if you misconfigured the routing on the transit gateway, in the source or destination attachments, or in the inspection VPC attachment, then traffic doesn't correctly route.

Review the following sections to identify and resolve your routing issue.

Check your source and destination attachment's route table configuration

To check whether the source and destination attachments have the correct route table associations, take the following actions:

• Verify that the attachments associate with different route tables from the inspection VPC attachment.
• Verify that the source and destination attachment's route tables contain a default static route to the inspection VPC attachment.
• Verify that the source and destination attachments don't propagate routes to the same route table that associates with these attachments.

For the inspection VPC attachment, verify that the route table it associates to contains static or propagated routes for CIDR blocks for both source and destination VPCs.

Check VPC route tables

For the inspection VPC, check whether your route table directs inspection traffic to the correct firewall endpoint. If you use Gateway Load Balancer endpoints, then verify that your load balancer endpoint subnet's routes point source and destination VPC CIDR blocks back to the transit gateway. If your routes direct traffic directly through the firewall, then verify that your firewall subnet's routes point source and destination VPC CIDR blocks back to the transit gateway.

For the destination and source VPCs, verify that the route tables contain routes that point to the transit gateway for remote VPC CIDR blocks.

Check connectivity and traffic flow to identify where traffic stops or drops

Take the following actions:

• If you turned on VPC Flow Logs, then check the flow logs to identify where your VPCs drop or reject traffic.
• If you turned on Transit Gateway Flow Logs, then check transit gateway flow log records to monitor traffic through attachments.
• Monitor the Amazon CloudWatch metric BytesDropCountBlackhole to identify routes that point to nowhere.
• Monitor the CloudWatch metric BytesDropCountNoRoute to identify packets with an invalid route.
• Test connectivity from the source to destination. Use ping, telnet, and traceroute to test connectivity from your source to destination traffic. For more information, see Using the ping, traceroute, and telnet commands on the ICDSOFT website.
• Compare the expected traffic path with the observed behavior.

Fix asymmetric routing issues

Your firewall might drop traffic because packets take different return paths.

To fix asymmetric routing issues, turn on appliance mode for stateful firewall inspection VPC attachment. Then, test both request and response packet paths.

Check security configurations

Take the following actions:

• Review security group rules for Amazon Elastic Compute Cloud (Amazon EC2) instances and firewalls based on your specific source and destination IP addresses.
• Check network access control list (network ACL) configurations for your transit gateway attachment, source, destination, and VPC endpoint subnets.
• Verify that firewall rules and policies allow the intended traffic.
• Confirm that port and protocol settings match your application's requirements.

Check firewall health and configuration

Take the following actions:

• Check your firewall health and endpoint status.
• Check firewall logs for dropped or rejected connections.
• Verify that your firewall rules are in the correct order. Rules at the top of the list override rules that are later in the list.
• Check whether your firewall exceeds its capacity limits and drops packets.

Get Gateway Load Balancer details

Note: If you don't see the Gateway Load Balancer configuration details in your AWS account, then it might be in a different account from your VPC endpoint.

If traffic doesn't route through your load balancer, then contact your firewall team for information about the load balancer configuration and status.

Test and validate your changes

To confirm that your changes fixed the routing issue and that traffic flows correctly, take the following actions:

• Perform end-to-end connectivity tests after you change configurations.
• Check traffic in your VPC flow logs.
• Verify that your application works as expected.
• Document what caused the issue and how you fixed it.
• Set up CloudWatch alarms to identify potential traffic routing issues.

Related information

What is AWS Transit Gateway for Amazon VPC?

What is AWS Network Firewall?

View transit gateway route tables using AWS Transit Gateway

Using CloudWatch alarms and Lambda to catch exceptional traffic

Centralized inspection architecture with AWS Gateway Load Balancer and AWS Transit Gateway