ecs cluster + asg

0

I have an ECS Cluster with 3 Nodes without Capacity Provider and therefore without ASG running for a long time. I have recently created a Capacity Provider and an ASG and I am raising new nodes with ASG to replace the old ones, I have put the old nodes in Drain mode and the Services have moved to the new ASG nodes, all good. The interesting thing is that I have 3 Target Group and Load Balancer so in each Target group a Registeres target is being raised on port 80 and this Target does not correspond to any of my services running on the nodes. Do you know what it could be?

asked a month ago128 views
1 Answer
1
Accepted Answer

it seems it is orphaned targets in your ECS cluster's target groups. Orphaned targets are instances that are registered with a target group but are not serving traffic for any running tasks or services.

some steps resolve the issue:

Go to the AWS Management Console, navigate to the EC2 service, and then select "Target Groups" from the sidebar. Check each target group associated with your ECS services and look for targets that do not correspond to any of your running tasks or services.

Double-check the configuration of your Auto Scaling Group (ASG) (https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-verify-scaling-activity.html)to ensure that it is correctly registered with the ECS cluster and that it is using the appropriate launch configuration or launch template.

Review the configuration of your ECS services to ensure that they are correctly configured to use the newly created capacity provider and ASG. Make sure that the desired task count and task definition are set correctly.

Log in to the ECS container instances running in your cluster and check their status. Verify that they are healthy and running the expected tasks. You can use ECS CLI or AWS CLI commands to retrieve information about the container instances and running tasks. (https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-health-checks.html)

Ensure that the load balancer associated with your ECS services is configured correctly and is targeting the appropriate target groups. Check for any misconfigurations or inconsistencies in the load balancer settings.

If you identify orphaned targets in your target groups, you can manually deregister them from the target group using the AWS Management Console or CLI. This will remove the unnecessary targets and prevent them from receiving traffic. )https://docs.aws.amazon.com/elasticloadbalancing/latest/application/delete-target-group.html)

If you continue to experience issues, you may want to consider reaching out to AWS Support for further assistance and troubleshooting.

profile picture
EXPERT
answered a month ago
profile picture
EXPERT
reviewed a month ago
  • Back with an update, I have found that every time the ASG adds a new instance, it creates a registered target in each target group associated to the ALB on port 80 and it remains unhealthy, I guess it is because of the EC2 health checks property that is always active in the ASG. My only temporary solution has been to delete the Registered target on port 80 in each Target Group manually after each scale out of the ASG.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions