How do I troubleshoot issues related to scheduled tasks in Amazon ECS?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

WhenWhen you use a scheduled task, Amazon EventBridge calls the RunTask API for Amazon ECS to run tasks on your behalf. Your scheduled Amazon ECS task might not run for the following reasons:

• You incorrectly configured the EventBridge time or cron expression.
• The EventBridge rule or schedule doesn't invoke the target.
• The RunTask API failed to run.
• The container instance exited because of application issues or resource constraints.
• The scheduled tasks failed because of capacity constraints.
• The scheduled tasks failed because of network connectivity issues.

Confirm that the EventBridge cron expression is configured correctly

To get the EventBridge cron expression, run one of the following AWS CLI commands.

If you use EventBridge rules, then run the following describe-rule command:

aws events describe-rule --name example-rule --region example-region

If you use EventBridge Scheduler, then run the following get-schedule command:

aws scheduler get-schedule --name example-schedule --region example-region

Note: Replace example-rule with your rule name, example-schedule with your schedule name, and example-region with your AWS Region.

In the command's output, check the EventBridge cron expression in the ScheduleExpression parameter. Verify that you set the rule schedule to the UTC+0 time zone.

Verify that the rule or schedule invokes the target

To check the Invocations and FailedInvocations Amazon CloudWatch metrics for the EventBridge rule, complete the following steps:

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose All metrics.
3. Choose Events.
4. Choose By Rule Name.
5. Select TriggerRules, Invocations, and FailedInvocations for the EventBridge rule that runs the Amazon ECS task.
6. Choose the Graphed metrics tab.
7. For Statistic, select SUM for each metric .

To check the performance of the EventBridge Scheduler, review the InvocationAttemptCount, TargetErrorCount, and InvocationDroppedCount CloudWatch metrics.

If you see FailedInvocations or InvocationDroppedCount data, then verify that the EventBridge AWS Identity and Access Management (IAM) role has permission to invoke your task.

If you see TargetErrorCount data, then there's an issue with the target invocation. Check whether your task started, started and immediately failed, or didn't complete.

Check whether the RunTask action failed

Complete the following steps:

1. Open the AWS CloudTrail console.
2. In the navigation pane, choose Event history.
3. For Lookup attributes, select Event name.
4. For Enter an event name, enter "run task".
5. In the time range filter, select the times that you expected your scheduled Amazon ECS task to run.
6. In the table, select the event that you want to view.
7. In the JSON event records, check for errorMessage or responseElements.failures.reason to confirm that the API didn't invoke the scheduled task.
   Note: You can find the task ID for the failed task in the responseElements.tasks.taskARN field.
8. Troubleshoot the API failure.

Check whether the container exited after the task ran

Even after the task successfully runs, Amazon ECS tasks might stop because of application issues or resource constraints. To resolve this issue, see How do I troubleshoot Amazon ECS tasks that stop or fail to start when my container exits?

Check for capacity constraint issues

Capacity constraint issues that cause tasks to fail occur when your cluster doesn't have enough compute resources, such as CPU and memory. Or, the issues occur when other workloads that run on the same cluster cause high resource usage. Restrictive task placement constraints or strategies might also reduce your available resources.

To monitor your resource usage, set up Container Insights on your Amazon ECS clusters. You can also set up CloudWatch alarms to notify you when you reach a resource usage threshold. To reserve resources for your tasks, set CPU and memory reservations. You might need to increase the desired count for the Amazon ECS service to scale out the cluster, or use cluster auto scaling.

Review existing task placement constraints, and adjust them to be less restrictive. Prioritize critical workloads, reschedule non-essential workloads, and test different placement strategies, such as spread, binpack, or random.

Use Amazon ECS capacity providers to reserve capacity for important workloads. Use Amazon Elastic Compute Cloud (Amazon EC2) Dedicated Instances for workloads that require isolated hardware. Use AWS Fargate for tasks that require available resources.

Check for network connectivity issues

To troubleshoot network connectivity issues for scheduled tasks, complete the following actions:

• Verify that the security group and network access control list (network ACL) rules allow the required inbound and outbound traffic for your tasks.
• Check that you correctly configured the internet gateway or NAT gateway for your virtual private cloud (VPC).
• Check your gateway route tables.
• Review your task definitions for environment variables with hardcoded IP addresses or hostnames. Verify that existing IP addresses or hostnames are correct, and you can reach them.

To test network connectivity from your container instances, complete the following steps:

1. Use SSH to connect to your Amazon EC2 instance.
2. To remotely run commands on your container, run the following execute-command command:

   aws ecs execute-command --cluster cluster-name --task task-id --container container-name --command "/bin/bash" --interactive

   Note: Replace cluster-name with your cluster name, task-id with your task ID, and container-name with your container name.
3. To test connectivity, run one of the following commands:

   curl -v http://example.com

   -or-

   nslookup example.com

   -or-

   nc -zv example.com 80

   Note: Replace example.com with your domain.
4. Check the DNS resolution and routing to identify network issues.

To simplify connectivity, run tasks on Fargate instead of Amazon EC2. With Fargate, you don't need to manage the underlying infrastructure. Or, use VPC endpoints to connect to services without an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.