Managing RDS Proxy connections when using Blue/Green Deployments with RDS or Aurora

By using Amazon RDS Proxy, you can allow your applications to pool and share database connections to improve their ability to scale. RDS Proxy makes applications more resilient to database failures by automatically connecting to a standby DB instance while preserving application connections. By using RDS Proxy, you can also enforce AWS Identity and Access Management (IAM) authentication for databases, and securely store credentials in AWS Secrets Manager.

Using RDS Proxy, you can handle unpredictable surges in database traffic. RDS Proxy establishes a database connection pool and reuses connections in this pool. This approach avoids the memory and CPU overhead of opening a new database connection each time. RDS Proxy queues or throttles application connections that can't be served immediately from the connection pool.

By using Amazon RDS Blue/Green Deployments, you can make and test database changes before implementing them in a production environment. A blue/green deployment creates a staging environment that copies the production environment. In a blue/green deployment, the blue environment is the current production environment. The green environment is the staging environment and stays in sync with the current production environment.

You can make changes to the RDS DB instances in the green environment without affecting production workloads. For example, you can upgrade the major or minor DB engine version, upgrade the underlying file system configuration, or change database parameters in the staging environment. You can thoroughly test changes in the green environment. When ready, you can switch over the environments to transition the green environment to be the new production environment. The switchover typically takes under a minute with no data loss and no need for application changes.

However, if you have RDS Proxy setup for your RDS/Aurora database and want to leverage the feature of Blue/Green Deployments, currently this isn't directly supported and you will observe an error while creating the Blue/Green Deployment. In this guide, we'll walk through the process of successfully managing RDS Proxy with RDS/Aurora databases while using Blue-Green Deployment.

[+] Limitations for blue/green deployments: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-considerations.html#blue-green-deployments-limitations

Prerequisites:

Before you get started, make sure you have the following prerequisites:

• An existing RDS/Aurora database supporting Blue/Green Deployment. For viewing the supported regions and DB engine versions for Blue/Green Deployments, kindly refer the Amazon RDS / Amazon Aurora documentation guide.
• RDS Proxy configured and running. For instructions, see Getting started with RDS Proxy documentation.

Error Information:

When trying to create a Blue/Green Deployment when RDS Proxy is already registered with the database, the below error prompt occurs on the RDS Console since RDS Proxy is not supported with Blue/Green Deployments:

Solution Overview:

In order to use RDS Proxy along with Blue/Green Deployments for RDS and Aurora, the workaround solution steps that you need to follow are outlined below:

Creating Blue/Green Deployment with RDS Proxy

1. De-register the RDS proxy target group associated to the concerned RDS instance/cluster.
2. Create the Blue/Green deployment.
3. Register the RDS Proxy target group and associate it with the blue instance/cluster. (Note: You will have to create a separate proxy if you want to connect to green instance/cluster as well)

Performing Major Version Upgrade

4. Perform RDS/Aurora engine upgrade in the green instance/cluster.

Switching over a blue/green deployment

5. De-register the RDS proxy target group associated to the concerned RDS blue instance/cluster.
6. Perform Blue/Green switchover.
7. Register the RDS Proxy target group and associate it with the blue instance/cluster.

Detailed Implementation Steps:

1. De-registering RDS Proxy Target Group

Before initiating the Blue-Green deployment, you must first de-register the existing proxy target group. Ensure the IAM role or user executing the command has the necessary permissions to de-register DB proxy targets. This usually includes the rds:DeregisterDBProxyTargets action. The AWS CLI command is as follows:

aws rds deregister-db-proxy-targets \
    --db-proxy-name my-proxy-name \
    --target-group-name default \
    --db-instance-identifier blue-instance-identifier

Documentation: Managing an RDS Proxy

[+] deregister-db-proxy-targets - https://docs.aws.amazon.com/cli/latest/reference/rds/deregister-db-proxy-targets.html

2. Creating Blue-Green Deployment

Create a new Blue-Green deployment environment for your RDS/Aurora database:

Using AWS CLI:

aws rds create-blue-green-deployment \
    --blue-green-deployment-name my-deployment \
    --source database-arn \
    --target-engine-version target-version

Using AWS Console:

1. Navigate to Amazon RDS Console
2. In the navigation pane, choose "Databases"
3. Select your source database (blue environment)
4. Choose "Actions" → "Create Blue/Green deployment"
5. Enter deployment identifier
6. Configure settings for the green environment
7. Choose "Create Blue/Green deployment"

Documentation: Creating a blue/green deployment

3. Registering RDS Proxy with Blue Instance/cluster

After creating the Blue-Green deployment, register the proxy target group with the blue instance:

Using AWS CLI:

aws rds register-db-proxy-targets \
    --db-proxy-name my-proxy-name \
    --target-group-name default \
    --db-instance-identifier blue-instance-identifier

Using AWS Console:

1. Navigate to Amazon RDS Console
2. Choose "Proxies"
3. Select your proxy
4. Go to "Target groups" tab
5. Select the target group and click on "Edit"
6. Choose the blue instance in Database section of Target group configuration
7. Click "Save Changes"

Documentation: Modifying an RDS Proxy

[+] register-db-proxy-targets - https://docs.aws.amazon.com/cli/latest/reference/rds/register-db-proxy-targets.html

4. Performing Major Version Upgrade

Execute the engine version upgrade on the green instance/cluster:

Using AWS CLI:

aws rds modify-db-instance \
    --db-instance-identifier green-instance-identifier \
    --engine-version new-version \
    --allow-major-version-upgrade \
    --apply-immediately

Using AWS Console:

1. Navigate to Databases
2. Select the green instance
3. Choose "Modify"
4. In "Database options", select new engine version
5. Choose "Continue"
6. Select "Apply immediately"
7. Choose "Modify DB instance"

Documentation: Upgrading a DB instance engine version

5. De-registering Blue Instance Proxy Before Switchover

Before switching to the green environment, de-register the proxy from the blue instance/cluster:

Using AWS CLI:

aws rds deregister-db-proxy-targets \
    --db-proxy-name my-proxy-name \
    --target-group-name default \
    --db-instance-identifier blue-instance-identifier

6. Performing Blue/Green Switchover

A switchover transitions the green environment to be the new production environment. When the green DB instance has read replicas, they are also transitioned. Before you switch over, production traffic is routed to the DB instance and read replicas in the blue environment. After you switch over, production traffic is routed to the DB instance and read replicas in the green environment. Kindly refer Switchover best practices before proceeding with the Blue/Green Switchover. You can refer the below steps to execute the switchover to the green environment:

Using AWS CLI:

aws rds switchover-blue-green-deployment \
    --blue-green-deployment-identifier deployment-identifier

Using AWS Console:

1. Navigate to Databases
2. Select the blue/green deployment
3. Choose "Actions" → "Switch over"
4. Review the switchover settings
5. Enter "confirm" in the confirmation field
6. Choose "Switchover"

Documentation: Switching a blue/green deployment in Amazon RDS

7. Registering RDS Proxy with New Primary/Blue Instance

After the switchover, register the proxy with the new primary instance:

Using AWS CLI:

aws rds register-db-proxy-targets \
    --db-proxy-name my-proxy-name \
    --target-group-name default \
    --db-instance-identifier new-primary-identifier

Using AWS Console: (Follow same steps as Step 3)

Important Considerations

1. Downtime Management:

   • Brief connection interruptions may occur during proxy de-registration and registration.
   • Plan these operations during maintenance windows.

2. Monitoring:

   • Monitor RDS Proxy connections before and after each step.
   • Verify application connectivity after each proxy registration.

3. Rollback Plan:

   • Maintain documentation of the original configuration.
   • Be prepared to reverse the process if needed.

4. Multiple Proxies:

   • Consider creating separate RDS Proxy endpoints for blue and green environments if needed.
   • Ensure proper resource cleanup after successful deployment.

Best Practices

1. Test the complete process in a non-production environment first.
2. Maintain detailed documentation of each step executed.
3. Have application teams ready during the switchover process.
4. Keep backup connection strings ready in case of issues.

For comprehensive information, refer to the AWS RDS documentation:

• RDS Proxy: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
• Blue-Green Deployments: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html