Share Your AWS re:Post Experience - Quick 3 Question Survey
Help us improve AWS re:Post! We're interested in understanding how you use re:Post and its impact on your AWS journey. Please take a moment to complete our brief 3-question survey.
How do I troubleshoot packet loss on my VPN connection?
I experience continual or intermittent packet loss and high latency issues on my VPN connection.
Resolution
Check resource utilization
High CPU, memory, or network bandwidth usage contribute to packet loss. Monitor the CPUUtilization, NetworkIn, NetworkOut, NetworkPacketsIn, and NetworkPacketsOut Amazon CloudWatch metrics. Check the resource utilization on both the source and destination hosts.
Check for network issues
To check for ICMP or TCP packet loss or latency, install the mtr network tool on the local on-premises machine. Also install mtr on an Amazon Elastic Compute Cloud (Amazon EC2) instance.
For Amazon Linux, run the following command:
sudo yum install mtr
For Ubuntu, run the following command:
sudo apt-get install mtr
For Windows, use WinMTR from SourceForge.
Note: WinMTR doesn't support TCP-based MTR.
Check whether there's packet loss inside the AWS VPN tunnel. Test the connection from the on-premises host that's using the tunnel to the private IP address of your Amazon EC2 instance. Also, test the connection from your on-premises customer gateway to the public IP address of each tunnel. If you can't use mtr on your customer gateway device, then use a host with the same internet connection as the customer gateway device.
Important: Obtain mtr results in both directions. The path between nodes on a TCP and IP address network can change when you reverse the direction.
Run the following test on a private IP address with ICMP:
mtr -b -c 50 -rw Host_Instance_Private_IP
Run the following test on a private IP address with TCP:
mtr -b -c 50 -rw -T -P 443 Host_Instance_Private_IP
Note: In the preceding commands, replace Host_instance_private_IP with the private IP address of your EC2 instance or on-premises host, and 443 with your port.
Run the following test on a public IP address with ICMP:
mtr -b -c 50 -rw AWS_Tunnel_Public_IP
Run the following tests on a public IP address with UDP:
mtr -b -c 50 -rw -u -P 500 AWS_Tunnel_Public_IP
mtr -b -c 50 -rw -u -P 4500 AWS_Tunnel_Public_IP
Note: In the preceding commands, replace AWS_Tunnel_Public_IP with the public IP address of the tunnel, and 500 and 4500 with your UDP ports.
Packet loss and increased round-trip time (RTT) on a single hop is expected. Packet loss and increased RTT are only issues if you view them on every hop. The following example output shows consistent loss that starts on hop 3 and continues until the final hop:
HOST: localhost Loss% Snt Last Avg Best Wrst StDev 1. 63.247.74.43 0.0% 10 0.3 0.6 0.3 1.2 0.3 2. 63.247.64.157 0.0% 10 0.4 1.0 0.4 6.1 1.8 3. 209.51.130.213 60.0% 10 0.8 2.7 0.8 19.0 5.7 4. aix.pr1.atl.google.com 60.0% 10 6.7 6.8 6.7 6.9 0.1 5. 72.14.233.56 50.0% 10 7.2 8.3 7.1 16.4 2.9 6. 209.85.254.247 40.0% 10 39.1 39.4 39.1 39.7 0.2 7. 64.233.174.46 40.0% 10 39.6 40.4 39.4 46.9 2.3 8. gw-in-f147.1e100.net 40.0% 10 39.6 40.5 39.5 46.7 2.2
The following example shows a test of the public IP address of a tunnel that uses UDP port 500. Between hops 1 and 4, the Loss% column shows a percentage loss. However, because there's no packet loss between hops 5 and 13, this loss doesn't qualify as significant packet loss. The loss between hops 14 and the final hop shows significant packet loss because it's a sustained loss that ends in 100% loss. The issue in this example occurs on hop 14.
sudo mtr -b -c 30 -rw -u -P 500 185.156.137.215 Start: 2024-08-28T18:59:49+0000 HOST: ip-10-100-20-25 Loss% Snt Last Avg Best Wrst StDev 1. |-- ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 2. |-- 240.1.84.6 73.3% 30 0.5 0.5 0.5 0.5 0.0 3. |-- 242.3.160.33 63.3% 30 1.4 1.0 0.6 1.4 0.3 4. |-- 241.0.6.67 63.3% 30 0.5 6.7 0.5 12.4 5.9 5. |-- 100.100.6.39 0.0% 30 0.5 3.1 0.5 27.7 6.2 6. |-- 100.100.72.6 0.0% 30 1.1 9.9 0.5 96.0 17.6 7. |-- 100.91.209.81 0.0% 30 11.5 20.6 11.1 127.3 24.3 8. |-- 100.100.6.107 0.0% 30 12.8 19.9 11.6 127.8 22.6 9. |-- 99.83.70.223 0.0% 30 12.3 18.1 11.7 117.0 19.2 10.|-- 100.100.92.209 0.0% 30 11.9 13.7 11.6 50.6 7.0 11.|-- 100.100.16.88 0.0% 30 13.2 14.0 11.9 33.1 3.8 12.|-- gw-as249.retn.net (87.245.240.41) 0.0% 30 13.2 13.2 11.8 19.9 1.5 13.|-- thw.sw01.loudltd.net (185.156.136.14) 0.0% 30 11.4 12.5 11.4 14.2 1.0 14.|-- no-ptr.pckear.com (185.116.108.124) 20.0% 30 14.8 13.2 12.5 14.8 0.5 15.|-- thw.sw01.loudltd.net (185.156.136.14) 40.0% 30 14.1 13.6 12.5 15.5 0.8 16.|-- thw.sw01.loudltd.net (185.156.136.14) 40.0% 30 16.5 14.5 13.3 16.5 0.8 17.|-- ??? 100.0% 30 0.0 0.0 0.0 0.0 0.0
Check for latency and routing issues
The traceroute tool shows you the path that data takes across the network. Use this information to identify where delays occur.
To install traceroute on Amazon Linux, run the following command:
sudo yum install traceroute
To install traceroute on Ubuntu, run the following command:
sudo apt-get install traceroute
For Windows, instead of traceroute, use the tracert tool. This tool is installed on all Windows machines by default. Note that tracert doesn't allow TCP trace. For full capabilities, you can install the tracetcp tool.
Run the following tests between the private and public IP address of your Amazon EC2 instance and your on-premises host.
Important: Obtain results in both directions. The path between nodes on a TCP and IP address network can change when you reverse the direction.
For Linux, run the following command:
sudo traceroute -T -p 80 Host_IP
Note: Replace Host_IP with the private IP address of your EC2 instance or on-premises host and 80 with your port. For better accuracy, the preceding example uses a TCP-based trace (T) to test the path of the traffic inside the tunnel. To test the path of the traffic over the internet, replace Host_IP with the public IP address of the tunnel.
You receive an output that looks similar to the following example:
traceroute -T -p 443 aws.com traceroute to aws.com (3.161.213.43), 30 hops max, 60 byte packets 1 SEA-1801842641.mshome.net (172.28.96.1) 0.363 ms 0.346 ms 0.362 ms 2 192.168.50.1 (192.168.50.1) 5.514 ms 5.508 ms 5.501 ms 3 172.27.232.1 (172.27.232.1) 45.179 ms 52.879 ms 52.871 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 server.yul62.r.ct.net (3.161.213.43) 57.452 ms 57.741 ms 62.604 ms
This example output shows intermediate hops with no response. Although the devices didn't respond on the requested 443 port, there's no issue because the destination still responds as expected.
For Windows, run tracert. Or, if you installed tracetcp, then run the following PowerShell command:
tracetcp.exe Host_IP:80
Note: Replace Host_IP with the private IP address of your EC2 instance or on-premises host and 80 with your port. For better accuracy, the preceding example uses a TCP-based trace (T) to test the path of the traffic inside the tunnel. To test the path of the traffic over the internet, replace Host_IP with the public IP address of the tunnel.
You receive an output that looks similar to the following example:
PS C:\Users\UserA> tracetcp.exe aws.com:443 Tracing route to 18.239.168.29 [server-18-239-168-29.bos50.r.cloudfront.net] on port 443 Over a maximum of 30 hops. 1 5 ms 7 ms 6 ms 192.168.50.1 2 41 ms 42 ms 44 ms 172.27.232.1 3 * * * Request timed out. 4 * * * Request timed out. 5 * * * Request timed out. 6 51 ms 43 ms 45 ms 241.0.11.143 7 62 ms 54 ms 43 ms 241.0.11.135 8 53 ms 53 ms 55 ms 240.4.8.24 9 52 ms 56 ms 53 ms 240.4.8.17 10 56 ms 59 ms 62 ms 240.64.239.162 11 57 ms 56 ms 61 ms 240.64.239.162 12 57 ms 55 ms 57 ms 240.64.239.162 13 58 ms Destination Reached in 54 ms. Connection established to 18.239.168.29 Trace Complete.
Test end-to-end TCP performance
Use hping3 to test end-to-end TCP packet loss and latency. This tool allows you to assess TCP performance directly between hosts.
To install hping3 on Amazon Linux, run the following command:
sudo yum --enablerepo=epel install hping3
To install hping3 on Ubuntu, run the following command:
sudo apt-get install hping3
To run hping3, run the following command:
hping3 -S -c 50 -V Host_IP
Note: Replace Host_IP with the private IP address of your EC2 instance or on-premises host, or with the public IP address of the tunnel.
The following example output shows a test against the public IP address of a VPN endpoint that uses port TCP 8443:
sudo hping3 -S -c 3 -p 8443 -V 34.233.155.9 using eth0, addr: 172.28.100.111, MTU: 1500 HPING 34.233.155.9 (eth0 34.233.155.9): S set, 40 headers + 0 data bytes len=44 ip=34.233.155.9 ttl=60 DF id=0 tos=0 iplen=44 sport=8443 flags=SA seq=0 win=26883 rtt=49.6 ms seq=2996814187 ack=1585541078 sum=22f7 urp=0 len=44 ip=34.233.155.9 ttl=60 DF id=0 tos=0 iplen=44 sport=8443 flags=SA seq=1 win=26883 rtt=49.6 ms seq=3555645549 ack=1207738370 sum=29a5 urp=0 len=44 ip=34.233.155.9 ttl=60 DF id=0 tos=0 iplen=44 sport=8443 flags=SA seq=2 win=26883 rtt=49.2 ms seq=4045561197 ack=742863285 sum=f78c urp=0
Identify issues with the network or application
Get simultaneous packet captures on both the EC2 instance and the on-premises host when the issue occurs. Use the packet capture to identify issues with the network or application.
To get a packet capture for Linux instances, use tcpdump.
To install tcpdump on Amazon Linux, run the following command:
sudo yum install tcpdump
To install tcpdump on Ubuntu, run the following command:
sudo apt-get install tcpdump
For Windows, instead of tcpdump, use Wireshark. To download the tool, see Download Wireshark on the Wireshark website.
The packet capture looks similar to the following example:
tcpdump -i any -nnvv port 500 and port 4500 tcpdump: listening on any, link-type EN10MB (Ethernet), capture size 65535 bytes 18:12:54.944933 IP (tos 0x0, ttl 63, id 43082, offset 0, flags [DF], proto UDP (17), length 212) 10.20.20.80.500 > 23.20.145.20.500: [udp sum ok] isakmp 1.0 msgid 00000000 cookie 19e7adeabfefca84→0000000000000000: phase 1 I ident: (sa: doi=ipsec situation=identity (p: #1 protoid=isakmp transform=1 (t: #1 id=ike (type=lifetype value=sec)(type=lifeduration value=7080)(type=enc value=aes)(type=keylen value=0080)(type=auth value=preshared)(type=hash value=sha1)(type=group desc value=modp1024)))) (vid: len=16) (vid: len=16) (vid: len=16) (vid: len=16) (vid: len=16) out slot1/tmm0 lis=$_ike_1061b_out_23.20.145.20 port=1.0 trunk= 18:12:54.946069 IP (tos 0x0, ttl 253, id 56278, offset 0, flags [none], proto UDP (17), length 152) 23.20.145.20.500 > 10.20.20.80.500: [udp sum ok] isakmp 1.0 msgid 00000000 cookie 19e7adeabfefca84→cfa493578af1f691: phase 1 R ident: (sa: doi=ipsec situation=identity (p: #1 protoid=isakmp transform=1 (t: #1 id=ike (type=enc value=aes)(type=keylen value=0080)(type=hash value=sha1)(type=group desc value=modp1024)(type=auth value=preshared)(type=lifetype value=sec)(type=lifedura tion value=7080)))) (vid: len=16) (vid: len=16) in slot1/tmm0 lis= port=1.0 trunk= 18:12:54.949988 IP (tos 0x0, ttl 63, id 43084, offset 0, flags [DF], proto UDP (17), length 256) 10.20.20.80.500 > 23.20.145.20.500: [udp sum ok] isakmp 1.0 msgid 00000000 cookie 19e7adeabfefca84→cfa493578af1f691: phase 1 I ident: (ke: key len=128) (nonce: n len=16 data=(e84ae917994501b5b4f4...1a547b3cf720e35995f5b0c69198c9f796b5d9f9)) (pay20) (pay20) out slot1/tmm0 lis=$_ike_1061a_in_10.20.20.80 port=1.0 trunk= 18:12:54.951858 IP (tos 0x0, ttl 253, id 56279, offset 0, flags [none], proto UDP (17), length 272) 23.20.145.20.500 > 10.20.20.80.500: [udp sum ok] isakmp 1.0 msgid 00000000 cookie 19e7adeabfefca84→cfa493578af1f691: phase 1 R ident: (ke: key len=128) (nonce: n len=32 data=(c5170686e88069e33323...4448aaf06cc37d4b32d67df97e275e5756a534d8)) (pay20) (pay20) in slot1/tmm0 lis=$_ike_1061a_in_10.20.20.80 port=1.0 trunk=
(Windows only) Turn off ECN
If explicit congestion notification (ECN) is on for your Windows instances, then you might experience packet loss or performance issues.
To check whether ECN is on, run the following PowerShell command:
netsh interface tcp show global
To turn off ECN, run the following PowerShell command:
netsh interface tcp set global ecncapability=disabled
Related information
Why does my application have latency and performance issues when I use Site-to-Site VPN?
Amazon EC2 instance-level network performance metrics uncover new insights
How do I troubleshoot low transfer speed on my Site-to-Site VPN?

Great article!
nice article!
Insightful
Very nice article!
Relevant content
- asked 2 years agolg...
- asked 2 years agolg...
- asked 2 years agolg...
- asked a year agolg...
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago