Investigating low-level TCP behavior of CloudFront-to-origin connections

Introduction

When you troubleshoot CloudFront in front of a custom origin (for example, an EC2 instance, an Application/Network Load Balancer, or an on-premises endpoint reached over the public internet), a lot of the interesting behavior happens below HTTP, at the TCP layer. Symptoms like intermittent 504 errors, retransmits, resets, or stalled transfers often trace back to how the origin-facing TCP connection is established and how packets are sized on the wire, not to anything visible in the access logs.

The most reliable way to answer "what is actually happening with our TCP connections to CloudFront?" is to capture the traffic.

This article walks through that methodology using one question as the worked example: what MTU does CloudFront use when it connects to a custom origin? It is a common point of confusion and easy to prove on the wire, so it makes a good template you can reuse for other low-level TCP investigations.

Why this matters

• Access logs and application metrics do not show handshake details, MSS/MTU negotiation, retransmits, or resets.
• Capturing on a purpose-built origin removes noise: only CloudFront can connect, caching is off, and you control the workload, so the flow you capture is the flow you care about.
• It removes guesswork. Instead of assuming a value or a behavior, you observe it directly. In the MTU example below, this is the difference between assuming jumbo frames apply and confirming the MSS CloudFront actually advertises.

Testing this in your own environment

1. Build the test architecture

The architecture is deliberately minimal so the only variable being measured is the CloudFront-to-origin TCP behavior:

• VPC with a public subnet and an internet gateway. The origin sits in the public subnet with a public IP so CloudFront reaches it directly over the internet as a custom origin (the same path as an origin behind an IGW).
• EC2 origin with a jumbo (9001) MTU interface. On boot, the instance sets its network interface MTU to 9001. This simulates a backend that prefers to use jumbo frames.
• A simple HTTP server that serves one large object (for example, a 10 MB file). A large object forces the origin to stream many data segments, which is measured in step 3.
• A CloudFront distribution with caching disabled. Using the managed CachingDisabled policy guarantees every request reaches the origin, so you always capture a fresh origin-facing connection.
• Locked-down ingress. The origin security group allows inbound TCP/80 only from the CloudFront origin-facing managed prefix list (com.amazonaws.global.cloudfront.origin-facing), so nothing but CloudFront can open a connection to the origin.
  • Note: We're using HTTPS and port 80 in a test environment to simplify the process, it's a best practice to use HTTPS for production workloads.
• SSM Session Manager for access. The instance runs the SSM agent with an instance profile granting AmazonSSMManagedInstanceCore, so you connect to run tcpdump without an SSH key or an open port 22.

Once the origin is running and the distribution is deployed, note the distribution domain name and connect to the instance over SSM:

aws ssm start-session --target <instance-id> --region <region>

2. Capture the MSS CloudFront advertises on the SYN

Connect to the origin with SSM Session Manager, start a capture for inbound SYNs, then drive a few requests through CloudFront. Add a random query string so caching (already disabled) is never a factor.

# On the origin (via SSM):
sudo timeout 50 tcpdump -ni any -vv 'tcp[tcpflags] & tcp-syn != 0 and dst port 80' > /tmp/syn.txt 2>&1

# Drive traffic through CloudFront (from somewhere other than the EC2 instance):
for n in 1 2 3 4 5 6; do
  curl -s -o /dev/null "https://<distribution-domain>"
  sleep 3
done

# Then inspect the SYN options:
grep -i 'mss' /tmp/syn.txt

Look for the mss <value> option in the SYN coming from a CloudFront POP IP. In testing, CloudFront advertised an MSS of 1440 (i.e., an effective ~1500-byte path MTU minus TCP/IP options), even though the origin interface was set to 9001.

3. Confirm the origin's actual segment sizes

Now prove the origin honors that MSS by capturing its outbound data segments while fetching the large object:

# On the origin (via SSM):
sudo timeout 55 tcpdump -ni any 'tcp and src port 80' > /tmp/data.txt 2>&1

# Fetch the large object a few times through CloudFront (from somewhere other than the EC2 instance):
for n in 1 2 3; do
  curl -s -o /dev/null "https://<distribution-domain>/bigfile.bin?nocache=$RANDOM$n"
  sleep 4
done

# Largest segment payloads the origin sent:
grep -oE 'length [0-9]+' /tmp/data.txt | awk '{print $2}' | sort -n | uniq -c | tail -10

Despite the 9001-byte interface MTU, the largest payloads cap at about 1460 bytes, which is consistent with the small MSS negotiated during the handshake.

Expected results

Conclusion

This test confirms that CloudFront-to-origin connections negotiate a standard ~1500-byte MTU and a small TCP MSS during the handshake, and that the origin honors that MSS. Jumbo frames on the origin do not carry over to the CloudFront session, so the segment sizes you see on the wire are driven by what CloudFront advertises, not by the origin's MTU.

More broadly, the value here is the methodology, not just this one result. By pairing a caching-disabled distribution with a locked-down origin and an on-box tcpdump capture, you get a clean, repeatable way to observe the exact packets CloudFront exchanges with your origin. Whenever you need to reason about low-level TCP behavior for traffic entering CloudFront, such as MSS and MTU negotiation, PMTUD black holes, retransmits, resets, or handshake failures behind intermittent 504s, stand up a harness like this one and capture the flow directly. Measuring the connection beats inferring it, and the packets will tell you what is actually happening.

Clean up

Delete the VPC, EC2 origin, and CloudFront distribution you created for the test once you have captured the results.