How do I troubleshoot high response times in Amazon ECS?

Resolution

Check your Amazon ECS service-level metrics

Higher CPU and memory utilization can cause high response latencies from your Amazon ECS tasks. Use Amazon CloudWatch metrics to view service-level CPU and memory utilization. If you activated CloudWatch Container Insights, then check container-level CPU and memory utilization. Container Insights with enhanced observability provides detailed task-level and container-level CPU, memory, disk, and network metrics with curated dashboards. If you didn't activate Container Insights, then see How do I configure CloudWatch Container Insights to monitor my Amazon ECS tasks and container instance?

If your application containers must perform CPU-intensive workloads, then upgrade to a larger task size that includes more virtual CPU (vCPU) and memory. Or, turn on Application Auto Scaling.

Check Amazon EC2 instance metrics for ECS tasks

Check the Amazon EC2 instance CPU and memory utilization for your Amazon ECS tasks that use the Amazon EC2 launch type.

Use CloudWatch metrics to monitor network performance of your Amazon EC2 instance. If the instance network bandwidth exceeds the available instance baseline bandwidth, then use a larger instance type with high baseline bandwidth.

Use the CPUUtilization CloudWatch metric to monitor instance CPU utilization. Also, log in to your instance to check the memory utilization. If CPU or memory utilization is high, then add more instances to the cluster to distribute tasks across multiple instances. Or, change the instance to a larger instance type.

Check the application logs to determine whether there's a high response latency from a downstream service. Use Amazon CloudWatch Application Signals to troubleshoot latency issues. To activate Application Signals on your Amazon ECS service, see Enable your applications on Amazon ECS.

Measure response latency

If the Amazon ECS service is behind an Application Load Balancer, then use the TargetResponseTime CloudWatch metric to measure the response time. This metric provides the average response time for requests that are routed through the Application Load Balancer. If the value is high, then there might be delays in the backend application or infrastructure.

To measure response latency directly from the container, log in to the application container using one of the following methods:

• For the Amazon EC2 launch type, connect to the underlying EC2 instance through SSH or AWS Systems Manager Session Manager. Then, run docker exec -it <container_id> /bin/bash to access the container. To find the container ID, run docker ps on the instance.
• For the AWS Fargate launch type, use ECS Exec to log in to the container because there is no underlying host for SSH access.
• For either launch type, the recommended approach is to use ECS Exec to log in to the container directly without SSH or host-level access. ECS Exec is available for both EC2 and Fargate launch types. ECS Exec provides secure access with IAM-based authorization and CloudTrail audit logging. For more information, see Monitor Amazon ECS containers with ECS Exec.

Then, run the following command to measure response latency:

curl -kso /dev/null -w "\n===============\n
| Connect: %{time_connect}\n
| App connect: %{time_appconnect}\n
| Pre-transfer: %{time_pretransfer}\n
| Time to First Byte (TTFB): %{time_starttransfer}\n
| Total: %{time_total}\n
| HTTP Code: %{http_code}\n===============\n" http://localhost:port/endpoint

Note: Replace the port with your container port and endpoint with your endpoint.

In the preceding command output, the Time to First Byte (TTFB) field shows the time for the first byte to arrive. The output also includes the server processing time. If the processing time is high, then the application takes longer to process the request.

Related information

How do I troubleshoot high CPU utilization on an Amazon ECS task on Fargate?

How do I monitor high memory utilization for Amazon ECS tasks on Fargate?

How do I calculate the total network usage for my Amazon EC2 instances and determine which instance used the most bandwidth within a date range?