Hi, I have encountered some strange behavior in my NLB that I would like your help with.
I have a NLB with two listeners (port 6379 and 6380) these listeners both forward requests to their TargetGroups and the TargetGroups to EC2 in an autoscaling group, these EC2 instances run Redis and I have other Instances (Web, API...) that use the NLB to connect to our Redis servers, so far so good, I've been using this for a few years without problems.
A few days ago I decide to add another lister to the NLB to port 6381 with a new TargetGroup, new AutoscalingGroup and of course another EC2 instance, this is another Redis server I will use to store Sessions, but the code to use this server was not ready yet so I set this new AutoscalingGroup to have 0 instances until my code was ready to use this new Redis server.
Suddenly I started to see that connections from my Web and API server to the old Redis instances on ports 6379 and 6380 were taking mins to complete, I was able to test this with a latency test from the redis-cli (redis-cli --latency -h mynlb.aws -p 6379) but also with the telnet command.
I tested several things to find why but so far why I know is that if instead of set the new AutoscalingGroup to 0 instances I add at least one instance the problem is gone, this not make any sense to me because again my code is using the old ports, I never changed it, this is a new port and infrastructure that should not interfere with the old configuration, also if instead of using the DNS name of the NLB I use the private IPs of the old instances then everything works. So the problem is when the traffic is send through the NLB
AWS OFFICIALUpdated a year ago
Hi, thank you for the information! So the third TG, is it a new AZ? What are the AZ settings on the NLB? Do you see anything funny in the metrics during this time? Are you sending traffic to this new listener port?
Hi, thanks for your answer so I am not sending traffic to the new port I am totally sure of that. About AZ and subnets I have 3 prive subnets one in each AZ and the NLB is connected to those, this is an old configuration. The new TG and AutoscalingGroup have the same AZ and subnet than the old Redis servers, I used the same. About the metrics on the NLB everything look normal, metrics from before and after the issue look exactly the same.
Is the new subnet listening on a new target port or one of the target ports that is already in use?
Sorry I don't understand well :) the new componentes (NLB listener on port 6381, TargetGroup and AutoScalingGroup) are using the existen AZs, subnets and SecurityGroups that I had before, there I don't have any restriction ACL or rule to restrict ports, in fact is open to all traffic
So you have two sets of listener ports. One would be the listener for the NLB and other would be listener for the target group. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html
I understand that you added a new listener port for NLB but what is that new target group listening on?