Question about Service VPC through which all the traffic goes ?

Based on your description and the provided topology, it seems you're trying to implement a centralized inspection architecture using AWS Transit Gateway and a firewall in a Service VPC. This setup is designed to route traffic between the Server-VPC and Web-VPC through the firewall in the Service-VPC for inspection.

The issue you're experiencing when manually creating the components could be due to several factors:

1. Transit Gateway Route Tables: Ensure that you have created separate route tables for each VPC attachment on the Transit Gateway. The route tables should be configured to direct traffic through the Service VPC (firewall) before reaching its final destination.

2. VPC Route Tables: Check that the route tables in each VPC (Server-VPC and Web-VPC) have routes pointing to the Transit Gateway for traffic destined for other VPCs.

3. Firewall Configuration: Verify that the firewall in the Service-VPC is properly configured to allow and inspect traffic between the Server-VPC and Web-VPC.

4. Transit Gateway Attachment Configuration: Make sure the Transit Gateway attachments for each VPC are correctly associated with their respective route tables.

5. Appliance Mode: For the Service VPC attachment on the Transit Gateway, ensure that "Appliance Mode" is enabled. This helps maintain flow symmetry for traffic inspection.

To troubleshoot this setup:

1. Use VPC Flow Logs to track the path of traffic between VPCs.
2. Check the Transit Gateway route tables to ensure routes are correctly propagated and pointing to the right attachments.
3. Verify the VPC route tables in each VPC to make sure they have the correct routes to the Transit Gateway.
4. Use packet captures or logging on the firewall to confirm if traffic is reaching it as expected.
5. Test connectivity between instances in different VPCs using tools like ping or traceroute to identify where traffic might be getting dropped or misrouted.

If you're still having issues, you may want to compare your manual setup with the CloudFormation template used in the lab to identify any discrepancies in the configuration.

Sources
VPC-to-VPC traffic inspection - AWS Prescriptive Guidance
Troubleshoot VPN routing issues | AWS re:Post
Networking 101 - Part 2 | AWS re:Post

Hello.

Reachability Analyzer is recommended for troubleshooting network routing.
By using Reachability Analyzer, you can isolate whether there is a problem with the VPC route table or the TransitGateway route table.
https://docs.aws.amazon.com/ja_jp/vpc/latest/reachability/how-reachability-analyzer-works.html

Transit gateway attachments should be in their own subnets.

Why do you have 5 attachments in the services VPC? What associations do you have on your TGW RTs?

Heres one way to do what your trying to acheive with GWLB https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/using-gwlb-with-tg-for-cns.html

Heres east to west with a Firewall which your looking at https://aws.amazon.com/blogs/networking-and-content-delivery/deploy-centralized-traffic-filtering-using-aws-network-firewall/

Sorry for typos.
Server-VPC-RT, Web-VPC-RT, FrontEnd-RT and BackEnd-RT are VPC route tables TGW-Service-RT is transist route table

thanks so much !!

I am also confused with the following statement from the AWS AI response above: "Ensure that you have created separate route tables for each VPC attachment on the Transit Gateway." In my understanding, I need to create only one transist gateway route table with the association with the Server-VPC and Web-VPC attachments.

Is that right ?

thanks !!

I used 10.1.99.0/27 for transist gateway service-vpc attachment. So the transist gateway default route sends the traffic to the ENI in service-vpc via this subnet. right ? Then the service-vpc internal router sends the traffic via the FrontEnd-RT in service-vpc ( in my case, it sends to ENI eth0). right ?

Anyway can I check the FrontEnd-RT from the cli via aws-cli ? thanks !!

If my web-ec2 in web-vpc sends traffic to server-ec2 in server-vpc via firewall in service-vpc, my understanding is as follows for the routing Web-VPC-RT --> TGW-Service-RT --> FrontEnd-RT --> eth0. If I capture packets on eth0, I am supposed to see the packets from web-ec2 based on the configuration I included above.

right ?

Unfortunately I did not. I do not know why?

thanks !!

If server and web vpc are associated with the same route table in TGW and the routes for the VPCs are in the route table then traffic will go directly between VPCs and not via the firewall.

The web and server VPCs need to be associated with a route table which only has a default Route 0.0.0.0/0 to the service VPC

The service VPC needs to be associated with a TGW route table with the populated routes from web and server VPC

This is Web VPC route table Web-VPC-RT Association 10.3.0.0/24 Routes 0.0.0.0/0 transist gateway 10.3.0.0/16 local

This the transist gateway route table TGW-Serivce-RT
Association Attachment-10.2.0.0/24 Association Attachment-10.3.0.0/24 Routes 0.0.0.0/0 Service-VPC

This is Service VPC route table FrontEnd-RT
Association 10.1.1.0/24 Association 10.1.99.0/27 Routes 0.0.0.0/0 ENI (eth0) 10.1.0.0/16 local

Based on the above table, I am supposed to the incoming traffic from web -> server on firewall eth0 interface, right ?

thanks !!

I have the following. When I propagated the web and the server attachments, the web and the server can reach each other. That tells me that the web and the server work ok via the transist gateway. After I removed the propagation from TGW-Service-RT, only left the default route pointing to service-VPC, the web stopped to reach the server. I ran tcpdump on eth0 and did not see any traffic from the web to the server. Can anyone help me to take a look at those two blue lines? Are they right ?

thanks a lot !!

Today I ran reachability analyzer to notice TGW-Service-RT is not used, I do not know why.

Anyone can help me on this ??

thanks !!

I just ran reachability analyzer to trace client --> server and I have the following:

Anyone can help me to understand why the default route is not taken ?

thanks a lot !!

I still could not figure out where the issues are. The following are the detailed configuration VPC and TGW attachments TGW route table forwarding to Service VPC VPC route table forwarding to FW interface

I ran tcpdump on the FW interface and did not see any incoming traffic from the Client.

Reachability Analyzer is as follows:

really appreciate anyone who can help to take a look.

thanks in advance !!!

My issue has been figured out. It was caused by source/dest check which should be disabled, as I used a linux for firewall.

thanks all !!