Why don't the Amazon EC2 instances in my Elastic Beanstalk environment send health data or pass ELB health checks?

Short description

Elastic Beanstalk enhanced health reporting relies on a health agent that collects health data on each instance. If communication between the agent and the Elastic Beanstalk endpoint fails, then the environment reports health data errors or missing health information.

Your Elastic Beanstalk environment might encounter health data or health check issues for the following reasons:

• You didn't activate enhanced health reporting in your environment.
• The health agent isn't running or can't communicate with the Elastic Beanstalk health endpoint.
• Your instances can't reach AWS service endpoints because of network connectivity issues.
• The instance profile doesn't have the required AWS Identity and Access Management (IAM) permissions to report health data.
• The application doesn't correctly respond to load balancer health checks.
• There are Amazon EC2 Auto Scaling or instance launch issues.

Resolution

Note: The following commands are for Linux instances.

Activate enhanced health reporting

If your environment uses basic health reporting, then instances don't report health data to Elastic Beanstalk. For instances to report health data, you must activate enhanced health reporting.

Note: If you use an .ebextensions configuration file that sets the health reporting system type, then the file overrides the console setting. Make sure that your .ebextensions file doesn't set the SystemType option to Basic in the aws:elasticbeanstalk:healthreporting:system namespace.

Check the Elastic Beanstalk health agent status

Complete the following steps:

1. Use SSH or Session Manager, a capability of AWS Systems Manager, to connect to your instance.

2. To check the health agent status, run the following command:

   sudo systemctl status healthd

3. If the health agent isn't running, then run the following command to restart it:

   sudo systemctl restart healthd

4. To review the health agent logs for errors, run the following command:

   sudo cat /var/log/healthd/daemon.log

5. To confirm that the instance can reach the Elastic Beanstalk health endpoint, run the following command:

   curl -v https://elasticbeanstalk-health.us-east-1.amazonaws.com/

   Note: Replace us-east-1 with your environment's AWS Region.

Check your network connectivity

If the health agent can't reach the Elastic Beanstalk health service endpoint, then check your network configuration settings.

Security groups

Check your instance's security group rules. The security groups must allow outbound traffic on port 443 to the Elastic Beanstalk health endpoint. They must also allow inbound traffic on the application from the load balancer security group.

Network ACLs

Check your instance subnet's network access control list (network ACL) rules. The network ACL must allow outbound HTTP traffic on port 443. The network ACL must also allow inbound traffic on the ephemeral ports (1024–65535) and traffic on the application port from the load balancer subnet.

Route tables

Check your route table configuration. If your instances are in a public subnet, then confirm that there's a route to an internet gateway. If your instances are in a private subnet, then confirm that there's a route to a NAT gateway. If your instances are in a private subnet without a NAT gateway, then create Amazon Virtual Private Cloud (Amazon VPC) endpoints for the following services:

• elasticbeanstalk-health.us-east-1.amazonaws.com
• sqs.us-east-1.amazonaws.com
• cloudformation.us-east-1.amazonaws.com
  Note: Replace us-east-1 with your environment's Region.

Check your IAM permissions

Make sure that the instance profile that's associated with your Elastic Beanstalk environment has the required permissions for the health agent to report data.

Update the instance profile role that's associated with Elastic Beanstalk to use the AWSElasticBeanstalkWebTier managed policy. For worker environments, use the AWSElasticBeanstalkWorkerTier managed policy. If you use a custom policy, then make sure that it includes the permissions from the AWSElasticBeanstalkWebTier managed policy.

If you recently changed the IAM role, then run the following command to restart the health agent and get the new credentials:

sudo systemctl restart healthd

Troubleshoot health check issues

If your application doesn't correctly respond to health checks, then the environment reports health issues.

To troubleshoot health check issues, complete the following steps:

1. Open the Elastic Beanstalk console.

2. Select your environment, and then choose Configuration.

3. Under Load balancer, choose Edit.

4. Under Health check, check the following settings:
   For Path, make sure that the endpoint returns an HTTP 200 status code.
   For Interval, make sure that the value is correct for your application. The default value is 15 seconds.
   For Timeout, make sure that the value is lower than the Interval value.
   Also, check Health threshold to configure consecutive successes and Unhealthy threshold to configure consecutive failures.

5. Choose Apply.

6. To locally test the health check path, connect to your instance and run the following command:

   curl -v http://localhost:5000/

   Note: Replace 5000 with the port your application listens on. If you have a custom health check path, then add the path. For example, add /health to the path.

7. If the health check path returns an error, then check the application status on the instance. To check whether the application process is running, run the following command:

   sudo systemctl status web

8. To confirm that the application listens on the expected port, run the following command:

   sudo ss -tlnp | grep 5000

   Note: Replace 5000 with the port that your application listens on.

9. To review the application logs for startup errors, run the following commands:

   sudo cat /var/log/web.stdout.log
   sudo cat /var/log/eb-engine.log

Troubleshoot Amazon EC2 Auto Scaling and instance launch issues

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

If your instances frequently launch and then terminate, then the health agent might not have enough time to initialize and send health data.

To check the Auto Scaling group activity history for launch failures, run the following describe-scaling-activities AWS CLI command:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name your-asg-name \
  --max-items 10

Note: Replace your-asg-name with the name of your environment's Auto Scaling group.

If instances terminate before the application fully starts, then complete the following steps to increase the health check grace period:

1. Open the Elastic Beanstalk console.
2. Select your environment, and then choose Configuration.
3. Under Rolling updates and deployments, choose Edit.
4. For Health check grace period, increase the value. For applications with long startup times, enter a value between 300 and 600 seconds.
5. Choose Apply.

To review the instance initialization logs for errors, run the following commands:

sudo cat /var/log/eb-engine.log
sudo cat /var/log/cfn-init.log
sudo cat /var/log/cfn-init-cmd.log

Related information

Health colors and statuses