- Newest
- Most votes
- Most comments
The ~130-second stall you're experiencing with concurrent database connections on fresh Fargate task launches is likely related to how the awsvpc network mode handles ENI initialization and connection establishment during the critical first moments of task startup.
When using awsvpc mode, each task receives its own elastic network interface (ENI) and private IPv4 address. During task launch, there's a brief window where the ENI is being attached and network resources are being fully initialized. While AWS doesn't publish specific internal timeout values, the ~130-second duration you're seeing suggests you may be hitting a connection-tracking or state-table timeout at the network layer.
The behavior you're observing—where concurrent connections stall but sequential connections work fine, and where everything works perfectly once the task is warm—points to a race condition during ENI initialization. When two connections attempt to establish simultaneously during this initialization window, the underlying network infrastructure may not yet be fully ready to handle multiple concurrent connection-tracking entries, causing one connection to hang until it times out.
For mitigation, your workaround of sequencing the database connections at startup rather than using Promise.all() is actually the recommended approach. You could also consider:
- Implementing a brief delay (even 1-2 seconds) before initiating concurrent connections to allow the ENI to fully stabilize
- Adding retry logic with exponential backoff specifically for startup connections
- Using a connection pooler like RDS Proxy, which can help smooth out connection establishment patterns
- If your application architecture permits, consider a health check or readiness probe that ensures network stability before proceeding with concurrent operations
The fact that your testing shows all individual components (DNS, TCP, TLS, single queries) work quickly on warm tasks confirms this is specifically about the concurrent connection pattern during the ENI initialization phase, not a configuration issue with your security groups, routing, or database setup.
Sources
Amazon ECS task networking for Amazon ECS Managed Instances - Amazon Elastic Container Service
AWSVPC mode - Amazon Elastic Container Service
Linux uses exponential backoff for TCP SYN retransmissions; with the default tcp_syn_retries=6, the total time is 1+2+4+8+16+32+64 = 127 seconds. This aligns closely with your observations.
The connection isn't merely slow; the SYN packets are being silently dropped (with no RST or ICMP response). You shouldn't be asking "why is it slow?" but rather "why is there no response to just one of the SYN packets?"
The fact that the issue was resolved by serializing the requests does not prove that concurrency was the root cause. Since the first request takes about 1.1 seconds, serialization might simply have introduced a one-second wait rather than eliminating concurrency.
Try inserting a 2–3 second sleep before the calls, while keeping Promise.all() as is.
- If it works → It’s a timing issue (waiting for the network path to be ready).
- If it doesn't work → Concurrency is indeed the cause.
This single test will confirm your hypothesis.
In Node.js 17 and later, the DNS resolution order defaults to "verbatim," meaning AAAA records may take precedence. If there is no IPv6 route in your VPC, the SYN packet enters a "black hole," leading to that 127-second timeout scenario. If the two connections selected different address families, it would explain why only one of them fails.
Try running it with this flag to see if the behavior changes:
node --dns-result-order=ipv4first
Workaround for the ECS Exec issue
VPC Flow Logs are the easiest option. By checking the flow logs for the task's ENI, you can see whether the SYN packet immediately following a cold start was ACCEPTED, REJECTED, or not recorded at all—without needing to access the container itself.
Update, following up on my own question — full investigation completed, and a working fix found.
Since asking this, I ran the raw-socket test I mentioned couldn't be done in isolation, plus several more variants, on real fresh task launches (not warm tasks via ECS Exec):
- Raw TCP socket (no driver): 6-77ms on a fresh task
- Bare
pg.Client(bypassing my ORM): 39-75ms - Bare
pg.Pool(matching my ORM's internal pooling config exactly): 99-102ms - Most tellingly: a brand-new
pg.Poolconnection fired at the exact same moment my real app's connection was actively failing/timing out — it succeeded in ~37ms while my app's own connection was still hung.
I also read my ORM's connection adapter source directly (@prisma/adapter-pg) — it's a thin wrapper, nothing unusual, just new pg.Pool() + pool.query().
So every layer I could test individually came back healthy, including at the literal moment of failure. This ruled out the ENI-warmup theory — a fresh connection at 9+ seconds into the task's life failed just as often as one at the very first instant.
The failure turned out to be genuinely intermittent, not deterministic — I built a "warm up with a disposable connection first" fix based on the finding above, and in one test run that itself failed 4 times in a row with the identical ~8s timeout, before a real connection then succeeded. No clean trigger I could isolate.
What actually fixed it: RDS Proxy. I put RDS Proxy in front of the database and pointed my app's DATABASE_URL at the proxy endpoint instead of RDS directly — zero application code changes. The exact same pre-investigation code, which had been failing intermittently, connected successfully and stayed healthy. This matches suggestion #3 from the earlier AI-generated answer here, and in my case it's looking like the real fix rather than just a mitigation, though I'm treating it as unconfirmed until it survives a few more days of redeployments given how intermittent the original issue was.
If anyone hits this same pattern (hard ~8s-or-so timeout, intermittent, every individual layer tests healthy in isolation) — RDS Proxy is worth trying before going deeper into ENI/network theories.
answered a month ago
Relevant content
- AWS OFFICIALUpdated 8 months ago
