Understanding AWS Accelerated Site-to-Site VPN Tunnel Lifecycle
Why tunnels sometimes don't recover after an endpoint replacement — and how to build resilient hybrid connectivity for semiconductor workloads.
Why this matters for Semiconductor and Hi-Tech Electronics
Chip-design and hardware-engineering organizations run some of the most connectivity-sensitive hybrid workloads on AWS. On-premises design centers routinely reach into AWS for:
- EDA burst compute — RTL simulation, place-and-route, and DRC/LVS signoff on HPC clusters
- License-server reachability — floating EDA licenses served from on-prem license managers to cloud compute
- Large dataset movement — PDKs, IP libraries, golden designs, and regression results between on-prem NAS/filers and cloud storage
For these workloads, a hybrid VPN that stays down for hours — let alone days — can stall a regression run, starve a verification farm of licenses, or slip a tape-out window. A common and surprising failure pattern: an AWS Site-to-Site VPN tunnel fails to re-establish after a routine AWS endpoint replacement, and stays down until someone manually intervenes. This article explains the full tunnel lifecycle, why that failure happens, and how to make hybrid connectivity resilient to it.
This article uses AWS Accelerated Site-to-Site VPN as the reference. The lifecycle concepts apply to standard Site-to-Site VPN as well; the acceleration layer (AWS Global Accelerator) introduces the NAT-Traversal behavior that makes one particular failure mode more likely.
How this differs from existing AWS guidance: AWS already documents the basics of tunnel endpoint replacement and accelerated-VPN troubleshooting (linked below). This article goes further for hybrid semiconductor / EDA workloads: the NAT-T (UDP 4500) return-path failure mechanics, the non-accelerated-tunnel diagnostic clue, and a resilience design (ECMP/BGP, Private IP VPN over Direct Connect, reachability-vs-tunnel-state monitoring). Start with the official references, then use this article for the deeper failure analysis:
- Understand why a Site-to-Site VPN tunnel endpoint was replaced — AWS Knowledge Center
- Troubleshoot Accelerated VPN issues — AWS Knowledge Center
- Scaling AWS VPN maintenance with tunnel endpoint lifecycle automation — AWS blog
1. Architecture overview
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ Customer Gateway │ │ AWS Global │ │ AWS VPN │
│ (CGW) │◄────►│ Accelerator │◄────►│ Endpoint │
│ On-prem IPsec │ Inet │ Static IPs, │ AWS │ (Regional) │
│ firewall / NGFW │ edge │ anycast routing │ back-│ │
│ (design center) │ │ e.g. 99.83.x.x │ bone │ │
└────────────────────┘ └────────────────────┘ └────────────────────┘
How Accelerated VPN differs from standard: the tunnel endpoints use static IP addresses from AWS Global Accelerator's pool, reached via anycast routing — customer traffic enters the AWS network at the nearest AWS edge Point of Presence and rides the AWS private backbone to the VPN endpoint in the target Region. This lowers and stabilizes latency versus traversing the public internet end-to-end — valuable when license checkouts and NFS/CIFS metadata calls are latency-sensitive.
Key consequence: because Global Accelerator sits in the path as a NAT, the tunnel uses NAT-Traversal (NAT-T), which is required and enabled by default for accelerated connections — and that detail is central to the failure mode below.
Prerequisites & constraints (read first):
- Acceleration is only supported on Site-to-Site VPN attached to a Transit Gateway — a virtual private gateway (VGW) does not support accelerated VPN.
- An accelerated VPN connection cannot be used with a Direct Connect public virtual interface.
- You cannot turn acceleration on/off on an existing connection. To change it, create a new VPN connection with acceleration set as desired, cut the customer gateway over, then delete the old connection.
- IKE negotiation for accelerated tunnels must be initiated from the customer gateway (see §7B).
⚠️ Certificate-based authentication caveat: Site-to-Site VPN connections that use certificate-based authentication might not be compatible with AWS Global Accelerator, due to limited support for packet fragmentation in Global Accelerator. If you require an accelerated VPN with certificate-based auth, your customer gateway device must support IKE fragmentation — otherwise, do not enable acceleration.
2. Tunnel establishment — the normal flow
IKEv2 brings the tunnel up in two phases. The port switches from UDP 500 to UDP 4500 once both sides detect NAT.
Phase 1 — IKE_SA_INIT (UDP 500)
Customer Gateway AWS VPN Endpoint
│ │
│ IKE_SA_INIT request (UDP 500) ─────────────────────►│
│ AES-256 · SHA2-256 · DH19 · NAT-D payloads │
│ │
│ ◄───── IKE_SA_INIT response (UDP 500) │
│ selected proposal + NAT-D response │
│ │
│ ★ NAT detected → all traffic moves to UDP 4500 │
│ │
Phase 2 — IKE_AUTH & Child SA (UDP 4500)
Customer Gateway AWS VPN Endpoint
│ │
│ IKE_AUTH request (UDP 4500) ───────────────────────►│
│ identity + PSK · ESP proposals · TS │
│ │
│ ◄───── IKE_AUTH response (UDP 4500) │
│ auth confirmed · CHILD_SA established │
│ │
│ ═══ TUNNEL IS NOW UP ═══ │
│ │
| Step | Direction | Port | Purpose |
|---|---|---|---|
| 1 | CGW → AWS | UDP 500 | Propose encryption params, detect NAT |
| 2 | AWS → CGW | UDP 500 | Select proposal, confirm NAT |
| 3 | CGW → AWS | UDP 4500 | Authenticate (PSK), propose ESP |
| 4 | AWS → CGW | UDP 4500 | Complete auth, establish data tunnel |
Key point: the UDP 500 → 4500 switch happens because Global Accelerator acts as a NAT. If UDP 4500 is impaired in either direction, the tunnel cannot complete or stay up — even though UDP 500 looked fine.
3. Tunnel maintenance — keeping it alive
| Mechanism | Port | Frequency | Purpose | If it fails |
|---|---|---|---|---|
| Dead Peer Detection (DPD) | UDP 4500 | ~10 s | Verify peer is alive | After the DPD timeout the peer is declared dead and the tunnel is torn down |
| NAT-T keepalive | UDP 4500 | ~20 s | Maintain NAT mapping on intermediate devices | NAT mapping expires → return traffic dropped |
| IKE SA rekey | UDP 4500 | hours | Rotate keys (forward secrecy) | SA expires → tunnel re-established |
DPD defaults (AWS Site-to-Site VPN): DPDTimeoutSeconds defaults to 30 seconds on new connections and 40 seconds when set via modify; the minimum is 30 seconds. DPDTimeoutAction defaults to clear (valid: clear | none | restart). With clear, AWS ends the IKE session on timeout and waits for the customer gateway to re-initiate. Setting the action to restart makes AWS re-initiate — a useful resilience lever (see §7B).
Semiconductor angle: NAT-T keepalive loss is insidious — the tunnel can look established while return traffic quietly stops, so a license-server heartbeat or an NFS mount can hang without an obvious "tunnel down" alarm. Monitor tunnel state and end-to-end reachability.
4. What happens during an AWS endpoint replacement
AWS periodically replaces VPN tunnel endpoints for software upgrades, patching, and hardware retirement. This is normal managed-service maintenance.
BEFORE DURING AFTER
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ Old endpoint │ │ Old endpoint │ │ NEW endpoint │
│ Has SAs ✓ │ │ TERMINATED │ │ NO SAs │
│ Has state ✓ │ └────────────────┘ │ Clean slate │
│ Static IP ────┼──────── IP PRESERVED ─────────────► Static IP │
└────────────────┘ └────────────────┘
Because the static (anycast-routed) IP is preserved across replacements, the customer gateway needs no reconfiguration. Expected recovery is under a minute:
| Step | What should happen | Timeframe |
|---|---|---|
| 1 | DPD responses stop → DPD timeout fires | ~30–40 s |
| 2 | CGW deletes old IKE SA + IPsec SA + session state | immediate |
| 3 | CGW re-initiates IKE_SA_INIT (UDP 500) to the same IP | seconds |
| 4 | Phase 1 + Phase 2 complete on the new endpoint | 1–5 s |
| 5 | Tunnel UP, traffic flows | < 1 minute |
5. The failure mode — when the tunnel stays down for days
Recovery depends on the customer gateway detecting the dead tunnel and clearing its stale state. When it doesn't:
CGW (on-prem firewall) AWS (new endpoint)
│ old SA still in memory │
│ stale flows in conn. table │
│ │
│ IKE_SA_INIT (UDP 500) ─────────────────────────────►│
│ ◄───── IKE_SA_INIT response (UDP 500) │
│ │
│ ✗ NAT-T switch to UDP 4500 fails (no return) │
│ ✗ or CGW matches STALE session, no renegotiate │
│ │
│ ━━ TUNNEL STAYS DOWN ━━ │
│ │
Two contributors commonly combine:
- Stale SA / session state on the customer gateway. The on-prem firewall keeps matching traffic to the old (invalid) Security Association or connection-table entry instead of negotiating a fresh SA. Behavior varies by firewall vendor and software version.
- UDP 4500 (NAT-T) return-path loss. Initial IKE on UDP 500 succeeds, but the transition to UDP 4500 fails in the return direction. Because acceleration can change the anycast entry PoP, an intermediate device (ISP CPE, on-prem NAT, or stateful firewall) may drop or fail to re-open the UDP 4500 mapping for the new path.
Possible causes for UDP 4500 return-traffic loss
- Stateful firewall / NAT device on-prem not re-creating the UDP 4500 mapping for the new flow
- ISP or intermediate device filtering or rate-limiting UDP 4500
- Asymmetric routing introduced by the anycast entry-PoP change
6. A useful diagnostic clue: does a non-accelerated tunnel survive?
If a standard (non-accelerated) Site-to-Site VPN to the same on-prem gateway recovers normally while the accelerated one does not, that strongly points to a NAT-T / UDP 4500 interaction rather than a general routing or firewall-policy fault. Standard VPN may negotiate without the same NAT-T dependency, so it sidesteps the broken 4500 return path. Use this as a fast triage signal.
7. Building resilience (recommendations)
A. Enable Tunnel Endpoint Lifecycle Control
This feature lets you manage when endpoint replacements happen instead of being surprised — you get advance visibility and a maintenance window to accept updates. Enable it per tunnel (JSON tunnel options; --skip-tunnel-replacement is a separate flag):
aws ec2 modify-vpn-tunnel-options \ --vpn-connection-id vpn-<vpn_identifier> \ --vpn-tunnel-outside-ip-address <TUNNEL_OUTSIDE_IP> \ --tunnel-options '{"EnableTunnelLifecycleControl": true}' \ --skip-tunnel-replacement
By default, enabling the feature on an existing VPN triggers an immediate endpoint replacement.
--skip-tunnel-replacementturns it on without one, so you can schedule the replacement during a maintenance window and validate recovery under controlled conditions. Repeat for every tunnel outside-IP on every VPN connection.
B. Make recovery self-healing (initiation & DPD)
- Keep the customer gateway as the IKE initiator. For accelerated VPN, IKE negotiation must be initiated from the customer gateway — AWS will not initiate. Ensure the CGW is configured to initiate and to re-initiate after a dead tunnel.
- Set
DPDTimeoutActiontorestartso the tunnel re-initiates automatically after a DPD timeout instead of waiting (clear). Use a DPD timeout at the documented minimum (≥30 s) appropriate for your gateway. StartupAction(valuesadd|start) controls whether AWS proactively initiates. Note:StartupAction=start(AWS-initiated) applies to non-accelerated connections — it does not override the "CGW must initiate" rule for accelerated VPN.
C. Fix stale-SA behavior on the customer gateway
Work with your firewall vendor so the gateway clears stale IKE/IPsec SAs for the peer when DPD declares the tunnel dead, and re-initiates negotiation. Where supported, configure the gateway as the IKE initiator.
D. Design for redundancy — don't depend on one tunnel
- Use both tunnels of each VPN connection (independent endpoints, separate Global Accelerator network zones) with a routing design that fails over between them.
- Prefer dynamic routing (BGP) so route withdrawal/re-advertisement drives fast failover.
- With Transit Gateway, use ECMP across multiple tunnels for both aggregate bandwidth (up to ~50 Gbps with multiple IPsec tunnels) and redundancy.
- For higher assurance, add a second VPN connection on a different customer gateway device, or complement VPN with AWS Direct Connect (VPN as backup).
- Eliminate the NAT-T failure mode entirely with Private IP VPN over Direct Connect. Running Site-to-Site VPN over a Direct Connect transit VIF with private IPs removes Global Accelerator (and its NAT/NAT-T) from the path — trading the accelerated public-internet path for private, dedicated connectivity.
- For latency-sensitive EDA license traffic, validate failover paths meet your round-trip and jitter budgets.
E. Monitor tunnel state and end-to-end reachability
- Alarm on Amazon CloudWatch namespace
AWS/VPN— key metricsTunnelState,TunnelDataIn,TunnelDataOut(perVpnId/TunnelIpAddress). - Also run application-level reachability checks (license checkout, NFS mount health) to catch silent NAT-T return-path loss that tunnel-state alarms miss.
- Enable Site-to-Site VPN tunnel logging (see §8) for IKE/DPD negotiation visibility.
8. Data to collect during the next occurrence
While the tunnel is down after a replacement, capture (on-prem side):
- IKE/IPsec SA table for the peer (stale SAs present?)
- Packet capture on UDP 500 and UDP 4500, both directions
MTR/ traceroute to the tunnel outside IP on UDP 500 and UDP 4500- Firewall logs for the IKE negotiation attempts
- Timestamp of the replacement vs. the moment traffic stopped
Enable Site-to-Site VPN tunnel logging on the AWS side for the definitive view: native IKE/IPsec/DPD logs are delivered to CloudWatch Logs on a best-effort basis at ~5-minute intervals, including TunnelIKEPhase1State, TunnelIKEPhase2State, and TunnelCGWNATTDetectionStatus (whether NAT-T was detected on the customer gateway). These fields quickly separate a stale-SA problem (fix on the gateway) from a UDP 4500 return-path problem (fix in the path/ISP/NAT device).
Author note: enabling tunnel logging on an existing tunnel briefly interrupts that tunnel — enable one tunnel at a time to preserve HA.
Summary
AWS Site-to-Site VPN endpoint replacements are routine and designed to be transparent — the static IP is preserved, and a healthy customer gateway recovers in under a minute by clearing stale state and re-initiating IKE. Tunnels that "stay down for days" almost always come down to (1) stale SA/session state on the on-prem gateway, and/or (2) UDP 4500 NAT-T return-traffic loss on the accelerated path. For semiconductor hybrid workloads where license reachability and dataset movement are on the critical path, the durable fix combines Tunnel Endpoint Lifecycle Control, CGW-initiated IKE with DPDTimeoutAction=restart, redundant tunnels/connections with BGP + ECMP (or Private IP VPN over Direct Connect to remove NAT-T entirely), and reachability-level monitoring — not just tunnel-state alarms.
References
- Accelerated AWS Site-to-Site VPN connections — rules & restrictions
- Enable Tunnel endpoint lifecycle control
- Scaling AWS VPN maintenance with tunnel endpoint lifecycle automation (blog)
- ModifyVpnTunnelOptions API — DPD timeout/action, StartupAction, lifecycle control
- Site-to-Site VPN logs — contents (IKE/DPD/NAT-T fields)
- Monitor VPN tunnels with Amazon CloudWatch (AWS/VPN metrics)
- Private IP VPN over AWS Direct Connect
