EKS connection string handling after Aurora PostgreSQL Global Database failover to secondary region

Question

Hello Team,
IHAC who has EKS cluster pods connecting to Aurora Postgresql Global Database Primary cluster endpoint. During an outage in the primary region, customer wants to point the EKS pods to the DB endpoint in secondary region. customer is familiar with process of failover from the database end.
Are there any best practices for switching EKS cluster pods from Primary cluster endpoint to secondary cluster endpoint? Do we have a circuit breaker solution that can be implemented.
Customer must point to the DB cluster endpoints and RDS proxy cannot be leveraged due to SSL Cert Requirement. 
Best practices for EKS database Connection handling during failover.

Thanks,
Karthik

Mina Gobrial · Accepted Answer

You can handle it by implementing a circuit breaker using a service mesh, you can follow these steps:
1. Deploy a service mesh such as AppMesh, Istio or Linkerd in your EKS cluster.
2. Configure the service mesh to route traffic to the primary Aurora PostgreSQL endpoint by default.
3. Configure a circuit breaker policy in the service mesh to detect when the connection to the primary endpoint fails.
4. When the circuit breaker detects a failure, it can automatically switch the traffic to the secondary Aurora PostgreSQL endpoint.
5. Once the primary endpoint is restored, the circuit breaker can switch the traffic back to the primary endpoint.

Another approach is to use DNS-based failover solution such as Route 53 to automatically switch the DNS record to the secondary region when the primary region is down. This allows the EKS pods to continue to connect to the database by using the same hostname, even when the database is switched over to the secondary region.

Here are some reference links that provide more information on implementing a circuit breaker using a service mesh in EKS:
* Istio documentation on circuit breaking: 
https://istio.io/latest/docs/concepts/traffic-management/#circuit-breaking

EKS connection string handling after Aurora PostgreSQL Global Database failover to secondary region

Relevant content