Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Site-to-Site VPN tunnel UP and passing traffic from on-premises to AWS, but zero data flows from AWS to on-premises

0

Issue: We have a Site-to-Site VPN conncetion (AWS <-> FortiGate 7.6.6) where the tunnel status shows UP on both tunnels, and traffic from the on-premises side to AWS works correctly and is confirmed flowing (TunnelDataIn shows bytes, VPC Flow Logs show accepted inbound traffic reaching our EC2 instance). However, traffic in the opposite direction (from our AWS VPC toward on-premises network) never leaves AWS. TunnelDataOut for the active tunnel remains at 0 bytes, even though:

  • Phase 1 and Phase 2 SAs established correctly (confirmed via AWS-side CloudWatch tunnel logs, including a healthy Phase 2 rekey with BOTH inbound and outbound CHILD_SA reported established)
  • VPC Flow Logs confirm our outbound packets are ACCEPTED and correctly routed toward the VGW (also tested with AWS Reachability Analyzer)
  • This is tested separate EC2 instances (including fresh AWS Linux installation), and multiple protocols (ICMP, UDB/SNMP), over multiple days

Questions:

  1. What could cause traffic to flow correctly in one direction but not the other, given that Phase 1/2 SAs are reported as healthy and established?
  2. VPC Flow Logs and Reachability Analyzer both indicate no issue at the route table / security group level. Is there any way to confirm, from the AWS side, whether packets actually reach the VGW and are presented for encapsulation? I'm trying to pinpoint whether the drop happens before the VGW, at the VGW's encapsulation step, or after?
  3. Are there any FortiGate-side commands worth running to further isolate this? Or does a persistently zero TunnelDataOut, combined with confirmed ENI-level ACCEPT, indicate the issue is entirely on AWS's side, since the customer gateway wouldn't yet be involved in a packet that was never actually sent?

Environment:

  • VPN Type: ipsec.1 and static routes
  • Two tunnels configured, but during testing, Tunnel 2 is temporarily disabled to rule out asymmetric routing issues
  • On-premises gateway: FortiGate 7.6.6

Things ruler out:

  • Route tables: Single route table per subnet, no overlapping routes, vgw correctly targeted
  • VPN connection's own static route: destination CIDR present and correct, state "available"
  • NACLs / security groups: Traffic accepted to both directions
  • VGW attached to the VPC correctly
  • VGW's route propagation enabled on Subnet's route table
  • IKE/IPsec parameters: confirmed against customer's FortiGate config directly
  • Phase 1 and Phase 2 SA establishment: confirmed healthy via AWS-side CloudWatch Logs
  • Forcing the active tunnel to actively re-initiate (StartupAction: start instead of add) had no effect
Topics
Networking & Content Delivery
Tags
VPC Virtual Private GatewayAWS Virtual Private Network (VPN)
Language
English

asked a month ago136 views

5 Answers
4

Having worked through sinmular issue with multiple customers, a persistently zero TunnelDataOut alongside an established Phase 2 SA almost always points to one of two FortiGate-side misconfigurations:

  • Missing FortiGate Static Route & Firewall Policy: In most cases I've seen, the issue is a missing static route on the FortiGate pointing the return traffic/destination subnet back to the IPsec tunnel interface, or a missing IPv4 Security Policy allowing inbound traffic from the AWS VPC CIDR over the VPN interface. If the FortiGate isn't properly set up to route/accept the return path, negotiation or return routing breaks silently.
  • Phase 2 Selectors (Proxy IDs) Mismatch: AWS static VPNs expect Phase 2 Proxy IDs to be broad (0.0.0.0/0). If specific subnets are defined in the FortiGate's Phase 2 configuration, Phase 2 may still report as "UP", but AWS will refuse to encapsulate packets from matching EC2 IPs before they ever leave the VGW.

Quick sanity check on FortiGate: Run a packet capture on the FortiGate CLI while initiating a continuous ping from your EC2 instance: diagnose sniffer packet any 'host <EC2-Private-IP>' 4 0 local

If zero packets arrive, check your Phase 2 Proxy IDs on the FortiGate. If packets arrive but get dropped, review your FortiGate static routes and firewall policies for the IPsec interface.

Dis you already run diagnose sniffer packet any 'host <EC2-Private-IP>' 4 0 local ?

EXPERT

answered a month ago

AWS
SUPPORT ENGINEER

reviewed a month ago

  • akoskena
    a month ago

    Thank you for the detailed answer, and for the sanity check suggestion.

    We've now confirmed both of your points on the FortiGate side:

    • Static route and firewall policy for this tunnel are present and correctly configured
    • Phase 2 selectors are broad (0.0.0.0/0), not narrowed - ruling out a selector mismatch

    We also ran the packet capture you suggested (diagnose sniffer packet any 'host <EC2-Private-IP>' 4 0 local) - zero packets arrive.

2

Based on the thorough troubleshooting you've already done, there are two remaining areas to investigate:

1. Verify the negotiated Phase 2 Security Association selectors (traffic selectors)

You mentioned Phase 2 selectors are configured as broad (0.0.0.0/0). However, the configured value and the actually negotiated value can differ. The VGW will only encapsulate packets that match a negotiated outbound Security Association. If the SA was negotiated with narrow selectors (even unintentionally), outbound traffic from AWS won't match and will be silently dropped - this is consistent with TunnelDataOut = 0.

On the FortiGate, check what was actually negotiated:

diagnose vpn tunnel list name <phase1-name>

Look for the proxy_id entries. You should see:

  • src: 0.0.0.0/0
  • dst: 0.0.0.0/0

Also check whether both inbound AND outbound SPIs exist:

diagnose vpn ipsec status

If you only see an inbound SA but no outbound SA, the VGW has no SA to encapsulate AWS-to-on-prem traffic.

2. Test bidirectional flow to isolate SA direction issues

As a diagnostic test: from on-premises, initiate a ping to the EC2 instance (which works). While that ping is actively running, simultaneously initiate a ping from the EC2 to on-premises. If the AWS-to-on-prem ping works ONLY while on-prem-to-AWS traffic is flowing, this confirms the outbound SA is only being created when the FortiGate initiates traffic.

3. Verify the VPC route table entry (sanity check)

You mentioned the route exists "via VGW's Route Propagation." Confirm:

  • The route state is active (not blackhole)
  • There is no more-specific route taking precedence

References:

AWS
SUPPORT ENGINEER

answered a month ago

  • akoskena
    a month ago

    Thank you for your suggestions, unfortunately the issue still remains unsolved.

    1. Verify the negotiated Phase 2 Security Association selectors (traffic selectors) We verified that broad proxy_ids are used (src: 0.0.0.0/0 / dst: 0.0.0.0/0) and that both inbound AND outbound SPIs exist.

    2. Test bidirectional flow to isolate SA direction issues While testing, we run ping to both directions simultaneously, but only on-prem-to-AWS traffic flows, and no packets are flowing towards on-premises.

    3. Verify the VPC route table entry (sanity check) We have confirmed that route states are active, and not blackhole, and that there are no more-specific routes taking precedence.

1

Hello.

As described in the documentation, TunnelDataOut counts data prior to encryption.
Therefore, a value of 0 for this metric suggests that the traffic may be stopping somewhere before it reaches the customer gateway.
https://docs.aws.amazon.com/vpn/latest/s2svpn/monitoring-cloudwatch-vpn.html

The bytes sent from the AWS side of the connection through the VPN tunnel to the customer gateway. Each metric data point represents the number of bytes sent after the previous data point. Use the Sum statistic to show the total number of bytes sent during the period.
This metric counts the data before encryption.

Is it possible that it's being blocked by the EC2 OS firewall or something similar?
I recommend using tcpdump or a similar tool on the EC2 instance to capture packets and examine their contents.

EXPERT

answered a month ago

EXPERT

reviewed a month ago

  • akoskena
    a month ago

    I've run tcpdump on the EC2 instance and confirmed the ICMP echo requests are actually transmitted out of the instance's NIC. I've also checked VPC Flow Logs for the same ENI, which show the traffic as ACCEPT, correctly matched against the route table. Based on checking the security group and ACL settings, they are also not dropping the packets. As a sanity check, and to rule out any EC2 OS firewall misconfigurations, I also recreated the EC2 from scratch (fresh AWS Linux EC2) and tested with that, with no luck. Reachability analyser has been also run from EC2 to target on-premise destination address, with Reachability status: Reachable, path stepping through EC2 instance, Network Instance, Security Group, ACL, propagated route on the route table, and VPN Gateway / VGW as the last step, which should confirm that the security groups/NACLS and route tables are not blocking the traffic.

    Other than that, is there anything else that I should check?

0
Accepted Answer

Thank you everyone for your answers.

After troubleshooting this from every angle, the issue was finally solved by moving the subnet and EC2 instance from Local Zone to Parent Region, and after that, the ping and VPN started working immediately.

The main issue was that even though the documentation does mention "The AWS Site-to-Site VPN is not available in Local Zones. Use a software-based VPN to establish a site-to-site VPN connection into a Local Zone." (https://docs.aws.amazon.com/local-zones/latest/ug/how-local-zones-work.html), this was misunderstood so that only the VPN connection and VGW can't be deployd within a Local Zone, and what was missed was that the subnet/EC2 instance within a Local Zone also cannot reach a VPN connection/VGW hosted in the parent region.

answered a month ago

0

Since you mentioned that the traffic is not reaching Customer Gateway, Please also check routes under the VPN configuration on AWS console. Thats most common setting i have seen being missed to confure in most cases.

AWS

answered a month ago

AWS
SUPPORT ENGINEER

reviewed a month ago

  • akoskena
    a month ago

    I have checked the following routes:

    • Route from EC2's subnet to the VGW existsts (via VGW's Route Propagation)
    • VPN connections's static route is set correctly
    • Reachability analyser has been run from EC2 to target on-premise destination address, with Reachability status: Reachable, path stepping through EC2 instance, Network Instance, Security Group, ACL, propagated route on the route table, and VPN Gateway / VGW as the last step

    Other than that, is there anything else that I should check?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content