How do I troubleshoot an Elastic Beanstalk environment that's in a Degraded or Severe state?

Short description

To troubleshoot an Elastic Beanstalk environment that's in a Degraded or Severe state, take one of the following actions:

• Use AI-powered analysis to identify and troubleshoot issues.
• Run the AWSSupport-TroubleshootElasticBeanstalk to automatically identify configuration issues.
• Manually troubleshoot your environment.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use AI-powered environment analysis to identify and troubleshoot issues

Prerequisites:

• Make sure that your environment meets the configuration requirements.
• Make sure that your environment's Amazon Elastic Compute Cloud (Amazon EC2) instance profile has the required AWS Identity and Access Management (IAM) permissions.
  Note: The AWSElasticBeanstalkWebTier, AWSElasticBeanstalkWorkerTier, and AWSElasticBeanstalkMulticontainerDocker managed policies include all required permissions.
• Make sure that you use a supported platform version.

AI analysis sends recent environment events, Amazon EC2 instance health status, and logs to Amazon Bedrock. You receive a root-cause summary with remediation steps. To run AI-powered analysis, you can use the Elastic Beanstalk console or the AWS CLI.

Note: The retrieve-environment-info AWS CLI command includes a presigned Amazon Simple Storage Service (Amazon S3) URL in the output that points to the analysis results.

After you fix the identified issues, reanalyze your environment to confirm that you resolved the issue.

Run the AWSSupport-TroubleshootElasticBeanstalk runbook to automatically identify configuration issues

Prerequisites: Make sure that your IAM user or role has the required permissions for the runbook. For more information, see Required IAM permissions in AWSSupport-TroubleshootElasticBeanstalk.

Run the AWSSupport-TroubleshootElasticBeanstalk runbook and configure the following input parameters:

• (Optional) For AutomationAssumeRole, enter the Amazon Resource Name (ARN) of the IAM role that allows the automation to perform actions. 
  Note: If you don't specify a role, then automation uses the permissions of the user that started the runbook.
• For ApplicationName, enter the name of your Elastic Beanstalk application.
• For EnvironmentName, enter the name of your Elastic Beanstalk environment.
• (Optional) If you have an active support case, then for AWSS3UploaderLink, enter the URL that AWS Support provided to upload the log bundle.

Troubleshoot the issues that the runbook identifies.

Manually troubleshoot your Elastic Beanstalk environment

Identify the cause of the health warning

To identify the cause of the health color and status, complete the following steps:

1. Open the Elastic Beanstalk console.
   Note: Make sure that you are in the correct AWS Region.
2. In the navigation pane, choose Environments, and then choose your environment.
3. Choose Events.
4. For Severity, select WARN to filter for warning events. Note the events that failed, the instance ID, and the underlying resource with issues.
5. On the Environment overview page, choose Causes to open the Health page. Or, in the navigation pane, choose Health.

For more information, see Enhanced health monitoring with the environment management console.

Troubleshoot the identified cause from the Health page

If the load balancer health checks fail, then see How do I troubleshoot Elastic Load Balancing health checks with Elastic Beanstalk? For other health check failures, see Basic health reporting. For enhanced health reports, see Enhanced health reporting and monitoring in Elastic Beanstalk.

For operations that take too long, use the Elastic Beanstalk event stream to identify the operation in progress. Or, connect to your EC2 instance, and then check the /var/log/eb-engine.log. Environment deployments and configuration updates might take longer than expected.

If there's an increase in 4xx and 5xx HTTP return codes, then check the access logs of the proxy server to identify the cause. Compare the access logs with the application logs to identify the error pattern.

Note: The proxy server logs are the access logs for Apache at /var/log/httpd/access_log, NGINX at /var/log/nginx/access_log, or IIS at C:\inetpub\logs\LogFiles.

For instances in a Severe state, troubleshoot the issue based on the cause in the warning. Instances can be in a Severe state because of an ongoing deployment, a failed health daemon, or high resource utilization. The warning state is temporary and transitions to green after you address the cause.

For a failed Elastic Beanstalk health daemon, connect to your instance, and then check /var/log/messages and /var/log/healthd/daemon.log to identify the cause. If you receive the "None of the instances are sending data" message, then see How do I resolve the "failed to communicate" error message that I get when my EC2 instances fail to communicate with Elastic Beanstalk?

Troubleshoot a failed application version deployment

If a new application version deployment fails, then complete the following steps:

1. Connect to the instance that failed.
2. Review /var/log/eb-engine.log for the failed step. Look for commands such as AppDeployPreHook, AppDeployEnactHook, or AppDeployPostHook.
3. Check your application logs, such as /var/log/web.stdout.log or /var/log/nginx/error.log, to identify the root cause.
   Note: Issues can occur because of missing environment variables, dependency or build failures, broken .ebextensions or Procfile entries, or a port binding issue.
4. After you fix the issue, redeploy the new version. Or, roll back to the last known working version.
5. Check the environment's Health page to confirm that the status is now Ok.

Troubleshoot a failed configuration update

A configuration update might fail after you change instance type, scaling settings, environment variables, load balancer settings, or .ebextensions. To troubleshoot this issue, connect to the instance that failed, and then check /var/log/eb-engine.log and /var/log/cfn-init.log for the failed step or .ebextensions command.

To check for resource-level errors, check the awseb-envID-stack stack in the AWS CloudFormation console. Look for UPDATE_FAILED or UPDATE_ROLLBACK_FAILED on the Amazon EC2 Auto Scaling group, launch template, or load balancer. To resolve these issues, make sure that you use valid option values and have the required IAM permissions. Also make sure that there are no syntax errors in .ebextensions or Procfile and that no setting conflicts with the current platform version.

After you fix the issue, redeploy the new version or roll back to the last known working version. Check the environment's Health page to confirm that the status is now Ok.

Identify and resolve out-of-band changes

Out-of-band changes occur if you or a user manually change your environment's underlying resources. These changes might result in environment health issues. Make changes to the resources in your environment only through Elastic Beanstalk.

To identify and resolve out-of-band changes, complete the following steps:

1. Open the CloudFormation console.
2. Choose the awseb-envID-stack.
   Note: Replace envID with your environment ID.
3. Run drift detection to list resources that have a live configuration that doesn't match the stack template.
4. Check your AWS CloudTrail Event history to identify the user that made the change, when it occurred, and through what API.
5. Update the resource to match the Elastic Beanstalk configuration. Or, reapply the change through an .ebextensions so that Elastic Beanstalk and CloudFormation stay in sync.