I'm struggling to identify the origin of some requests to my NAT Gateway. Followed the instructions for past posts to get the Cloudwatch logs, understood the issue were inbound requests, and the following query shows me a sample of what's causing it:
filter (srcAddr like /18.161.*/ or srcAddr like /3.163.*/ ) and dstAddr like '<nat-gateway-ip>' | limit 10
Those IPs belong to CLOUDFRONT and AMAZON services, as per described on the query.
The AMAZON requests (srcAddr like /3.163.*/) has a pkt-src I can't find on my network (will keep trying to find it).
The CLOUDFRONT is the most misterious one. pkt-src address is the src-addr IP itself, which belongs to AWS, the destination is the nat gateway, the ENI is from the NAT gateway, and it ends there - I have no clue what is the origin and intended destination on the network.
Has anyone faced something similar before?
Yes, the thing is that the pkt-srcaddr is exactly the same as the srcaddr. I do know we have way more ECS tasks spinning up due to the use of a tool called Dagster to run our syncs so I'm starting to consider that the issue is the downloads from ECR. Another thought I had was traffic between AZs as I only have one NAT in us-west-2a and containers spread across us-west-2a and us-west-2b.