How do I troubleshoot issues related to scheduled tasks in Amazon ECS?

5 minute read
1

I scheduled my Amazon Elastic Container Service (Amazon ECS) task to run periodically. However, my Amazon ECS task isn't activated. I'm not getting execution logs or the history of the tasks in the cluster.

Resolution

When you use a scheduled Amazon ECS task, Amazon EventBridge calls the RunTask API to Amazon ECS to run the tasks on your behalf.

Your scheduled Amazon ECS task might not be invoked due to the following reasons:

  • The Amazon EventBridge time or cron expression is configured incorrectly.
  • The EventBridge rule or schedule doesn't invoke the target.
  • The RunTask API failed to run.
  • The container exit due to application issues or resource constraints.

Confirm the EventBridge cron expression is configured correctly

To get the EventBridge cron expression, run the following AWS Command Line Interface (AWS CLI) command:

If Eventbridge rules is used
$ aws events describe-rule --name "example-rule" --region example-region 

If Eventbridge scheduler is used
$ aws scheduler get-schedule --name "example-schedule" --region example-region

In the output of the command, view the configured EventBridge cron expression in the parameter ScheduleExpression. Be sure that you set the schedule for the rule in the UTC+0 time zone.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Check whether the rule doesn't invoke the target

To view the rule performance, use the Amazon CloudWatch metrics generated by EventBridge. Invocation datapoints indicate that the target was invoked by the rule. If FailedInvocations data points are present, then there is an issue invoking the target. FailedInvocations represent a permanent failure and might be the result of incorrect permissions or a misconfiguration of the target.

To review the CloudWatch metrics for the EventBridge rule, do the following:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Metrics, and then choose All metrics.
  3. Choose Events.
  4. Choose By Rule Name.
  5. Select the TriggerRules, Invocations, and FailedInvocations metrics for the EventBridge rule that's configured to run the ECS task.
  6. Choose the Graphed metrics tab.
  7. For all metrics listed, select SUM for Statistic.

If FailedInvocations data points are present, there might be an issue related to target permissions. Be sure that EventBridge has access to invoke your ECS task. Verify that the EventBridge AWS Identity and Access Management (IAM) role has the required permissions. For more information, see Amazon ECS EventBridge IAM Role.

Check whether the schedule doesn't invoke the target

To view the rule performance, use the Amazon CloudWatch metrics generated by EventBridge. InvocationAttemptCount datapoints indicate that the EventBridge Scheduler is attempting to invoke your schedules. If TargetErrorCount data points are present, then there is an issue with the target invocation. InvocationDroppedCount represent a permanent failure and might be the result of incorrect permissions or a misconfiguration of the target.

To review the CloudWatch metrics for the EventBridge rule, do the following:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Metrics, and then choose All metrics.
  3. Choose Scheduler.
  4. Choose Schedule Group Metrics.
  5. Select the InvocationAttemptCount, TargetErrorCount, and InvocationDroppedCount metrics for the EventBridge rule that's configured to run the ECS task.
  6. Choose the Graphed metrics tab.
  7. For all metrics listed, select SUM for Statistic.

If InvocationDroppedCount data points are present, there might be an issue related to inadequate target permissions. Be sure that EventBridge has access to invoke your ECS task. Verify that the EventBridge AWS Identity and Access Management (IAM) role has the required permissions. For more information, see Amazon ECS EventBridge IAM Role.

Check whether the RunTask action ran

To verify if the RunTask API failed to run, search in AWS CloudTrail event history for RunTask. Look within the time range of when the scheduled ECS task was expected to be invoked.

To find if the scheduled task wasn't invoked because the RunTask action failed, do the following:

  1. Open the AWS CloudTrail console.
  2. In the navigation pane, choose Event history.
  3. In the Event history page, for Lookup attributes, select Event name.
  4. For Enter an event name, enter RunTask.
  5. Choose the time range in the time range filter based on when the scheduled ECS task was expected to run.
    Note: The preset values for the time range are 30 minutes, 1 hour, 3 hours, and 12 hours. To specify a custom time range, choose Custom.
  6. From the results list, choose the event that you want to view.
  7. To view the JSON event record, scroll to Event record on the Details page.
  8. Look for errorMessage or responseElements.failures.reason elements in the JSON event record.
    These elements in the JSON event record display the reason for the scheduled ECS task not being invoked.

For examples of RunTask API failure reasons and their causes, see API failure reasons in Amazon ECS.

Check whether the container exited after the task ran

The Amazon ECS tasks might be stopped even after the task runs successfully due to application issues or resource constraints. For more information, see How do I troubleshoot Amazon ECS tasks that stop or fail to start while my container exits?

Related information

How do I troubleshoot Amazon ECS tasks that stop or fail to start while my container exits?

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago