Hi,
I'm using terraform to create infrastructure for two environments: develop and production.
Both environments consist of a self-hosted kubernetes cluster on EC2 instances, and a self-managed database on an EC2 instance.
The develop env has all these in private subnets behind a NAT GW and a network load balancer. There are three target groups, one for http, one for https traffic pointing to the cluster and one for the protocol of the database. There are a few Route53 alias records pointing to our network load balancer and the target groups are associated to the right auto scaling groups. The cluster and the database is reachable from the public internet (this is intentional for the time being). This setup works very well.
The problem is when I tried to reproduce the same setup for the production environment, the database was sometimes unreachable, more not than yes, and when it wasn't, the connection was just hanging. The only thing different are the names, like environment name etc., the configuration is pretty much the same. I can't figure out why it works in one case and not in the other.
I've disabled cross-zone load balancing on both load-balancers so when I execute the dig command on the develop database record, I only get one IP address as it should be because of the disabled setting. But that isn't the case with the production NLB as I get 3, as much as the number of associated subnets. It's as though the cross load balancing setting is on even if it says it isn't.
Has anyone experienced inconsistent behavior like this?
In the end, I had to disassociate the production database from the production NLB target group, put it in a public subnet and create an A record just for it.
Hi, thanks for your comment. I realize how cross-zone load balancing works and agree with what you said however I didn't have it turned off right away. If you read my question, the exact same setup was used in both environments with cross-zone load balancing turned on and in spite of that I observed a different behavior which I can't explain.