Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

NLB PROXY Protocol stalls connections to server-speaks-first TCP backends (MySQL) - handshake completes, then ~5s silent gap before header is forwarded

0

What we're seeing

We have a Network Load Balancer with PROXY Protocol v2 enabled, in front of a TCP backend that speaks a "server speaks first" wire protocol (MySQL, via haproxy and separately via Percona's MySQL Router - same result both ways). New connections reliably stall: the TCP three-way handshake completes normally in under 100ms, but no data - not even the PROXY protocol preamble itself - gets forwarded to the target for several seconds afterward. If the client's own connect/read timeout is shorter than that delay (a 5s timeout is common for many MySQL client libraries), the client aborts before the PROXY header, and therefore the backend's own handshake, ever arrives.

This effectively makes NLB PROXY Protocol unusable with any server-speaks-first TCP protocol (MySQL, and presumably SMTP/FTP/similar) whenever the client's timeout is on the order of a few seconds.

Setup

  • NLB: internet-facing, dualstack, target-type ip, provisioned via the AWS Load Balancer Controller (v3.4.0) from a Kubernetes Service in EKS (region us-east-1, IPv6-only pod networking)
  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
  • Backend tested both as haproxy (accept-proxy on the frontend bind) and, separately, as MySQL Router directly - identical stall either way, which rules out the backend software as the cause.

Packet capture (isolated on a single-target debug NLB, no load-balancing ambiguity)

14:54:15.063831 IP6 <client>.7291 > <target>.3307: Flags [S], seq ..., win 64800, ...
14:54:15.063846 IP6 <target>.3307 > <client>.7291: Flags [S.], seq ..., ack ..., win 62503, ...
14:54:15.154602 IP6 <client>.7291 > <target>.3307: Flags [.], ack 1, win 507, ...
                                        <<< ~5 second silent gap, zero bytes either direction >>>
14:54:20.159837 IP6 <client>.7291 > <target>.3307: Flags [.], seq 1:101, ack 1, win 507, length 100
14:54:20.159837 IP6 <client>.7291 > <target>.3307: Flags [F.], seq 101, ack 1, win 507, length 0
14:54:20.159845 IP6 <target>.3307 > <client>.7291: Flags [.], ack 101, win 122, ...
14:54:20.160455 IP6 <target>.3307 > <client>.7291: Flags [P.], seq 1:86, ack 102, win 122, length 85
14:54:20.160536 IP6 <target>.3307 > <client>.7291: Flags [F.], seq 86, ack 102, win 122, ...
14:54:20.250874 IP6 <client>.7291 > <target>.3307: Flags [R], seq ..., win 0
14:54:20.250998 IP6 <client>.7291 > <target>.3307: Flags [R], seq ..., win 0
  • Handshake (SYN / SYN-ACK / ACK): ~90ms, healthy.
  • Then nothing for ~5.0 seconds.
  • At almost exactly the client's configured --connect-timeout=5, 100 bytes arrive (consistent in size with a PROXY protocol v1 text header for IPv6 addresses), immediately followed by a FIN from the same side.
  • The target replies with 85 bytes (consistent with MySQL's own initial handshake/greeting packet) - but by then the real client has already aborted, so it's moot.
  • Client-side error: ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 110 (ETIMEDOUT).

What we think is happening

The delay sits specifically between handshake completion and any payload being forwarded - exactly where the PROXY protocol preamble is supposed to be injected, immediately and unconditionally per the PROXY protocol spec. Our working theory: this NLB's PROXY protocol implementation doesn't forward/inject the header until it observes data flowing in a particular direction, which never happens on its own for a server-speaks-first protocol

  • the client is waiting for the server's greeting, the server is waiting for the PROXY header. The stall only breaks once something external forces data to move (here, the client's own timeout firing).

Workaround

Switched to the NLB's Client IP Preservation target-group attribute (preserve_client_ip.enabled=true) instead of PROXY Protocol - avoids the issue entirely since no header injection is needed (the real source IP rides on the packet itself via DNAT-only forwarding).

Question

Has anyone else run into this specific stall with NLB PROXY Protocol in front of a server-speaks-first TCP protocol? Is this documented/expected behavior of NLB's PROXY protocol implementation, or does it look like a bug worth a formal support case? Any pointers to existing documentation or open issues covering this would be appreciated.

Topics
Networking & Content DeliveryContainers
Tags
Amazon VPCNetwork Load BalancerElastic Load BalancingAmazon EKS Distro
Language
English

asked 6 days ago60 views

2 Answers
3
Accepted Answer

Your packet capture analysis is spot on. This is a known architectural behavior of NLB: the NLB waits for the first bytes of the client payload before it injects the PROXY protocol v2 header. With server-speaks-first protocols like MySQL or SMTP, this creates a TCP deadlock that only breaks when a timeout forces traffic.

Your workaround—disabling PROXY protocol and relying on Target Group Client IP Preservation (preserve_client_ip.enabled=true)—is the correct and widely accepted standard solution for this exact scenario.

It is still worth opening an AWS support case. While support will likely confirm it works as designed, filing it as a feature request adds weight to the demand for native server-speaks-first support with PROXY protocol on NLBs.

Reference for the IP preservation configuration: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation

EXPERT

answered 6 days ago

EXPERT

reviewed 6 days ago

1

The behavior you're observing appears to be related to a known limitation with how PROXY Protocol v2 interacts with certain AWS networking components. While your specific scenario involves NLB directly, there is documented guidance indicating that PROXY Protocol v2 is not supported in certain AWS service mesh configurations, which suggests broader compatibility considerations with this protocol version in AWS networking infrastructure.

For server-speaks-first protocols like MySQL and SMTP, the timing of when the PROXY protocol header is injected becomes critical. The 5-second delay you're seeing between handshake completion and header forwarding is problematic because these protocols expect the server to send its greeting immediately after the TCP handshake completes, but the server is waiting for the PROXY header first.

Your workaround of using Client IP Preservation instead of PROXY Protocol is actually the recommended approach for this type of scenario. Client IP Preservation uses a different mechanism that doesn't require header injection, making it more suitable for server-speaks-first protocols where timing is critical.

If you continue to experience issues or need PROXY Protocol specifically for your use case, you should open a case through the AWS Support Center at https://console.aws.amazon.com/support/ to add correspondence about this behavior. When doing so, include your packet captures and configuration details, as this level of technical detail will help the support team investigate whether this represents expected behavior given the protocol constraints or if there are additional configuration options that might help your specific scenario.
Sources
App Mesh connectivity troubleshooting - AWS App Mesh

answered 6 days ago

EXPERT

reviewed 5 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content