CICD failover setup and alert configuration

Hello.

It is possible to set approval actions in CodePipeline.
By setting this action in the DR CodePipeline, you can wait for deployment until approval is received.
Once you have confirmed that the application deployed in the primary CodePipeline is running successfully, you can approve the deployment.
https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals.html

If primary code not working correctly then i want a message that dr is handling traffic.

For the above, you can use Route53's failover routing to route to an application that is still working properly.
https://aws.amazon.com/jp/blogs/networking-and-content-delivery/creating-disaster-recovery-mechanisms-using-amazon-route-53/

Hello

Here is how you can achieve this with AWS native services :

1. Use AWS CodePipeline to create two pipelines linked to the main branch:

• Primary pipeline: triggers automatically on every code push to deploy new code immediately.
• Disaster Recovery (DR) pipeline: configured with a manual approval action that pauses the pipeline awaiting human approval before deployment.

2. Use Amazon CloudWatch metrics, alarms, synthetics or custom health checks to monitor the primary deployment’s stability and uptime for the desired period. Synthetic canaries are a good option as they allow testing the application on a regular vbasis, as seen by end users. Synthetics can generate alarms. You would want to trigger an alarm when the application has been up and running for the required amount of time. This alarm can be used to trigger (thru Amazon SNS, for instance, the DR pipeline start).

After confirming the primary pipeline is working correctly for that period, manually review and approve the DR pipeline’s deployment in the CodePipeline console, making use of AWS CodePipeline’s built-in manual approval action, which sends notifications via Amazon SNS to alert approvers.

The pipeline will wait at this manual approval stage until a user approves or rejects the deployment.

If the primary pipeline is found not working correctly, configure CloudWatch Synthetics alarms to trigger a notification (via SNS, for example) stating that DR is handling traffic, and do not proceed with DR pipeline deployment.

This approach ensures:

1. The primary pipeline deploys code immediately and automatically.
2. The DR pipeline deploys only after a waiting period and manual approval, preventing premature or bad deployments.
3. Notifications and monitoring keep track of the health/status of the primary deployment and DR readiness.
4. Manual control over DR deployment minimizes risk

How to setup cloudwatch synthetic alarm to inform the application ha been running for. the correct amount of time :

Create a CloudWatch Synthetics Canary:

• Go to the CloudWatch console and create a new Canary.
• Write or use a script to simulate user interactions or API calls to your application, ensuring it checks for availability and expected behavior.
• Schedule the Canary to run at a regular interval (e.g., every 5 minutes) to continuously verify application health.

Use Canary Metrics:

• The Canary emits metrics such as SuccessPercent or FailedRuns.
• A successful run indicates the application is working as expected.

Create a CloudWatch Alarm to capture stability :

• Create a CloudWatch alarm based on the Canary's success metric.
• Configure the alarm to trigger if the SuccessPercent remains at or near 100% for the last 24 hours, which means the application has been running OK continuously. -This can be done by setting the alarm to evaluate the metric over a 24-hour period (e.g., 288 data points if the Canary runs every 5 minutes) and raising the alarm if success stays high

Best regards

Serge