Understanding RDS Proxy Database Connection Scaling: Why Are There Too Many Open Connections During Peak Hours?

Question

Hi everyone,

I'm using AWS Lambda functions that connect to an RDS database through RDS Proxy. I manage connections following Sequelize’s best practices as they've guided here: [Using sequelize in AWS Lambda](https://sequelize.org/docs/v6/other-topics/aws-lambda/)

My current RDS Proxy configuration is similar to:
```
{
  "TargetGroups": [
    {
      "DBProxyName": "my-rds-proxy",
      "TargetGroupName": "default",
      "Status": "available",
      "ConnectionPoolConfig": {
        "MaxConnectionsPercent": 80,
        "MaxIdleConnectionsPercent": 15,
        "ConnectionBorrowTimeout": 120,
        "SessionPinningFilters": []
      }
    }
  ]
}
```
I observed that the dashboards indicate that client connections are being managed correctly. I expected that once a client connection is closed, RDS Proxy would return the corresponding database connection to the pool for reuse. The pool should maintain idle connections up to the limit defined by:
> max_connections * MaxConnectionsPercent * MaxIdleConnectionsPercent

For example, in my case, the underlying RDS instance has a max_connections setting of 1289. Based on the calculation (1289 * 0.8 * 0.15), I expected the maximum number of idle connections to be around 154. When combined with the active client connections (around 15 at peak), the total would be roughly 169 connections at any given time (a small discrepancy can be accepted).

However, during peak hours, I observe up to 600 database connections in the Dashboards, even though the number of max Client connections during the peak hours lies around 15. SHOW PROCESSLIST; outputs the same number of Database connections, with most showing a "Sleep" state, while only a handful are actively processing queries.

My questions are:
1. What exact mechanism or internal heuristics does RDS Proxy use to decide how many underlying database connections to maintain?
2. Why might it scale up to 600 connections even when my configuration seems to indicate a much lower combined limit of active and idle connections?
3. Are there best practices or additional configuration tweaks to better control the pool size if necessary?

Any insights, clarifications, or pointers to further documentation would be greatly appreciated. Thanks in advance for your help!

re:Post Agent · Answer

RDS Proxy uses complex internal mechanisms to manage database connections, which can sometimes lead to behavior that may seem counterintuitive. While your configuration settings provide guidelines, RDS Proxy may scale connections beyond these limits based on various factors.

1. RDS Proxy's connection management:
RDS Proxy uses advanced algorithms to determine the number of database connections to maintain. It considers factors such as current workload, historical patterns, and anticipated demand. The proxy aims to optimize performance and reduce connection latency by keeping a sufficient number of connections ready.

2. Scaling beyond expected limits:
The high number of connections (600) you're observing during peak hours, despite lower client connections, could be due to several reasons:

- Connection pooling: RDS Proxy may be keeping connections open in anticipation of future requests, even if they're not actively used.
- Session pinning: Certain types of database operations can cause connections to be "pinned" to specific sessions, preventing their immediate release back to the pool.
- Traffic patterns: If your application experiences consistent traffic, the proxy might keep more connections open to optimize for this pattern.
- Lambda concurrency: High concurrency in Lambda functions might lead the proxy to open more connections to handle potential load, even if not all are actively used simultaneously.

3. Best practices and configuration tweaks:

- Review and adjust the minimum and maximum connection settings in your RDS Proxy configuration.
- Monitor CloudWatch metrics for your RDS Proxy and RDS instance to understand connection usage patterns.
- Ensure your Lambda functions are properly reusing database connections when possible.
- Check the idle client timeout settings and adjust if necessary to reduce the number of open idle connections.
- Consider implementing your own connection monitoring on the client side to better understand actual database activity from the requester's standpoint.

Remember that while high connection counts may seem concerning, RDS Proxy is designed to manage these connections efficiently. If you're not experiencing performance issues, the current behavior might actually be optimal for your workload. However, if you're concerned about resource usage, you may want to experiment with adjusting the configuration parameters and closely monitor the results.  
**Sources**  
[RDS Proxy causing a consistent increase in DB Connections from Lambda | AWS re:Post](https://repost.aws/questions/QU2hiJKCLUQcq5ApSro0ixMQ/rds-proxy-causing-a-consistent-increase-in-db-connections-from-lambda)  
[Persistent High Connection Counts in RDS with Proxy: Seeking Community Insights and Solutions | AWS re:Post](https://repost.aws/questions/QUOpSHrH3YTCyL8UBEjRIpFQ/persistent-high-connection-counts-in-rds-with-proxy-seeking-community-insights-and-solutions)  
[RDS Sessions are increasing rapidly | AWS re:Post](https://repost.aws/questions/QUiZ-JkpfHThiR2rAqfCAssQ/rds-sessions-are-increasing-rapidly)

Sab · Answer

Hello ,

What is the value of Idle client connection timeout

Regards
Sabarish

Sab · Answer

I have seen this happening, however my MaxIdleConnectionsPercent was of a higher value. I tried reducing this and there was not much of a difference in behavior.   The traffic pattern of my app is unpredictable. So, I monitored the app for some time and there were no issues because of the connections not being released quickly. Generally, after 24 hours connections are released

Understanding RDS Proxy Database Connection Scaling: Why Are There Too Many Open Connections During Peak Hours?

Add your answer

Relevant content