By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Why do Batch jobs create new ECS Task Definition revisions?

0

I have a nightly process that is scheduled through EventBridge with a target of "AWS Batch" and an API of "SubmitJob". Everything works fine but every time it runs it creates a new Task Definition revision in ECS. After running for the past several months I now have hundreds of "active" revisions in my ECS service.

Is this normal? What am I doing wrong? How can I just get AWS Batch to use an existing revision instead of creating a new one every time?

Example CloudTrail event that is causing the new Task Definition:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA2Z2RLHZ2J36RVXOUQ:aws-batch",
        "arn": "arn:aws:sts::11111111111111:assumed-role/AWSServiceRoleForBatch/aws-batch",
        "accountId": "11111111111111",
        "accessKeyId": "ASIA2Z2RLHZ2LQC24AQF",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA2Z2RLHZ2J36RVXOUQ",
                "arn": "arn:aws:iam::11111111111111:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch",
                "accountId": "11111111111111",
                "userName": "AWSServiceRoleForBatch"
            },
            "attributes": {
                "creationDate": "2024-10-25T07:45:15Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "batch.amazonaws.com"
    },
    "eventTime": "2024-10-25T07:45:37Z",
    "eventSource": "ecs.amazonaws.com",
    "eventName": "RegisterTaskDefinition",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "batch.amazonaws.com",
    "userAgent": "batch.amazonaws.com",
    "requestParameters": {
        "family": "my-family",
        "taskRoleArn": "arn:aws:iam::11111111111111:role/my-role-iamrecsschedulerole-AGXL8WKlilZ8",
        "executionRoleArn": "arn:aws:iam::11111111111111:role/my-role-iamrecsexecutionrole-b5lHaNSVLm4U",
        "networkMode": "awsvpc",
        "containerDefinitions": [
            {
                "name": "default",
                "image": "222222222222.dkr.ecr.us-east-1.amazonaws.com/my-image:aaabbbbbcccc",
                "cpu": 0,
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-group": "/lh/batch/sg/jobs",
                        "awslogs-region": "us-east-1",
                        "awslogs-stream-prefix": "my-project"
                    },
                    "secretOptions": []
                }
            }
        ],
        "requiresCompatibilities": [
            "FARGATE"
        ],
        "cpu": "1024",
        "memory": "2048",
        "tags": [
            {
                "key": "AWSBatchServiceTag",
                "value": "batch"
            }
        ],
        "runtimePlatform": {
            "cpuArchitecture": "X86_64",
            "operatingSystemFamily": "LINUX"
        }
    },
    "responseElements": {
        "taskDefinition": {
            "taskDefinitionArn": "arn:aws:ecs:us-east-1:11111111111111:task-definition/my-family:259",
            "containerDefinitions": [
                {
                    "name": "default",
                    "image": "222222222222.dkr.ecr.us-east-1.amazonaws.com/my-image:aaabbbbbcccc",
                    "cpu": 0,
                    "portMappings": [],
                    "essential": true,
                    "environment": "HIDDEN_DUE_TO_SECURITY_REASONS",
                    "mountPoints": [],
                    "volumesFrom": [],
                    "logConfiguration": {
                        "logDriver": "awslogs",
                        "options": {
                            "awslogs-group": "/lh/batch/sg/jobs",
                            "awslogs-region": "us-east-1",
                            "awslogs-stream-prefix": "my-project"
                        },
                        "secretOptions": []
                    },
                    "systemControls": []
                }
            ],
            "family": "my-family",
            "taskRoleArn": "arn:aws:iam::11111111111111:role/my-role-iamrecsschedulerole-AGXL8WKlilZ8",
            "executionRoleArn": "arn:aws:iam::11111111111111:role/my-role-iamrecsexecutionrole-b5lHaNSVLm4U",
            "networkMode": "awsvpc",
            "revision": 259,
            "volumes": [],
            "status": "ACTIVE",
            "requiresAttributes": [
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
                },
                {
                    "name": "ecs.capability.execution-role-awslogs"
                },
                {
                    "name": "com.amazonaws.ecs.capability.ecr-auth"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role"
                },
                {
                    "name": "ecs.capability.execution-role-ecr-pull"
                },
                {
                    "name": "ecs.capability.secrets.ssm.environment-variables"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
                },
                {
                    "name": "ecs.capability.task-eni"
                }
            ],
            "placementConstraints": [],
            "compatibilities": [
                "EC2",
                "FARGATE"
            ],
            "requiresCompatibilities": [
                "FARGATE"
            ],
            "cpu": "1024",
            "memory": "2048",
            "runtimePlatform": {
                "cpuArchitecture": "X86_64",
                "operatingSystemFamily": "LINUX"
            },
            "registeredAt": "Oct 25, 2024, 7:45:37 AM",
            "registeredBy": "arn:aws:sts::11111111111111:assumed-role/AWSServiceRoleForBatch/aws-batch",
            "enableFaultInjection": false
        },
        "tags": [
            {
                "key": "AWSBatchServiceTag",
                "value": "batch"
            }
        ]
    },
    "requestID": "2cd25ad5-3383-42c3-a03f-0e6caccb45df",
    "eventID": "c2109255-8278-4349-bd7d-1161eee7ca38",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "11111111111111",
    "eventCategory": "Management"
}
Topics
ComputeContainersDevOps
Tags
AWS BatchAmazon Elastic Container ServiceContainersDevOps
Language
English
profile picture
RNHurt
asked 2 months ago142 views
3 Answers
2

Hello.

I set up EventBridge Scheduler and AWS Batch in my AWS account to run a Batch job every 5 minutes, but the revision number of the ECS task definition did not increase from 1.
Is it possible for you to share what settings you are using on EventBridge?

The configuration below is a sample EventBridge scheduler that I created.

aws scheduler get-schedule --name batch-test
{
    "ActionAfterCompletion": "NONE",
    "Arn": "arn:aws:scheduler:ap-northeast-1:111111111111:schedule/default/batch-test",
    "CreationDate": "2024-10-25T00:35:42.536000+00:00",
    "Description": "",
    "FlexibleTimeWindow": {
        "Mode": "OFF"
    },
    "GroupName": "default",
    "LastModificationDate": "2024-10-25T00:35:42.536000+00:00",
    "Name": "batch-test",
    "ScheduleExpression": "rate(5 minutes)",
    "ScheduleExpressionTimezone": "Asia/Tokyo",
    "State": "ENABLED",
    "Target": {
        "Arn": "arn:aws:scheduler:::aws-sdk:batch:submitJob",
        "Input": "{\n  \"JobDefinition\": \"arn:aws:batch:ap-northeast-1:111111111111:job-definition/getting-started-wizard-job-definition:1\",\n  \"JobName\": \"getting-started-wizard-job-test\",\n  \"JobQueue\": \"arn:aws:batch:ap-northeast-1:111111111111:job-queue/getting-started-wizard-job-queue\"\n}",
        "RetryPolicy": {
            "MaximumEventAgeInSeconds": 86400,
            "MaximumRetryAttempts": 185
        },
        "RoleArn": "arn:aws:iam::111111111111:role/service-role/Amazon_EventBridge_Scheduler"
    }
}

Also, if you search for "RegisterTaskDefinition" in CloudTrail's event history, you can search for events that update the task definition, so you may be able to identify the cause from there.
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/tutorial-event-history.html

profile picture
EXPERT
Riku_Kobayashi
answered 2 months ago
profile picture
EXPERT
Oleksii Bebych
reviewed 2 months ago
  • RNHurt
    2 months ago

    My configuration is similar but the Target:Input includes container overrides. Maybe that is causing the new task definitions to be defined?

    {
    "ActionAfterCompletion": "NONE",
    "Arn": "arn:aws:scheduler:us-east-1:111111111111:schedule/default/my-job",
    "CreationDate": "2024-03-01T22:03:43.340000-05:00",
    "Description": "",
    "FlexibleTimeWindow": {
        "Mode": "OFF"
    },
    "GroupName": "default",
    "LastModificationDate": "2024-10-01T09:44:45.471000-04:00",
    "Name": "my-job",
    "ScheduleExpression": "rate(5 minutes)",
    "ScheduleExpressionTimezone": "US/Eastern",
    "State": "ENABLED",
    "Target": {
        "Arn": "arn:aws:scheduler:::aws-sdk:batch:submitJob",
        "Input": "{\"JobDefinition\":\"my-job\",\"JobName\":\"my-job-name\",\"JobQueue\":\"my-queue\",\"ContainerOverrides\":{\"Command\":[\"bundle\",\"exec\",\"rails\",\"db:my-job\"]}}",
        "RetryPolicy": {
            "MaximumEventAgeInSeconds": 86400,
            "MaximumRetryAttempts": 0
        },
        "RoleArn": "arn:aws:iam::111111111111:role/batch-sg-iamrecsschedulerole-AGXL8WKlilZ8"
    }
}
  • RNHurt
    2 months ago

    That was a good tip to look for "RegisterTaskDefinition" in CloudTrail. I found the events that were causing the problem quickly, and indeed it appears to be the "aws-batch" user.

    Assuming that it's the container overrides that are causing the duplicate Task Definitions, it's interesting that none of the overrides actually appear in the new Task Definitions. I'm stumped!

  • Riku_Kobayashi EXPERT
    2 months ago

    I tried running AWS Batch by setting "ContainerOverrides" in my AWS account, but no new revisions were created in the ECS task definition. Is it possible for you to share the details of the "RegisterTaskDefinition" event recorded in CloudTrail?

  • RNHurt
    2 months ago

    @Riku_Kobayashi I updated the question with more details. Hopefully that helps resolve the issue.

  • Riku_Kobayashi EXPERT
    2 months ago

    Isn't the AWS Batch job definition updated every time with "RegisterJobDefinition" before "RegisterTaskDefinition"? If the job definition is updated every time, the task definition will also be updated every time. In other words, the following flow may occur in your AWS account every time. I think the cause is "RegisterJobDefinition", so I need to look for this event in CloudTrail to determine who is updating it. For example, do you have some kind of process that updates the job definition when batch processing ends?

    RegisterJobDefinition -> SubmitJob -> RegisterTaskDefinition
1

The behavior you're experiencing with AWS Batch creating new ECS Task Definition revisions for each job run is not typical and suggests there might be a misconfiguration in your setup.

Normally, AWS Batch should use an existing Task Definition revision rather than creating a new one each time a job runs. The Task Definition specifies the container properties, resource requirements, and other configurations for your batch job. It should remain consistent unless you intentionally update it.

There are a few potential reasons why this might be happening:

  1. Your EventBridge rule or AWS Batch job definition might be configured to create a new Task Definition revision each time it runs. Check your EventBridge rule configuration and Batch job definition to ensure they're not set to generate new revisions.

  2. If you're using Terraform or another Infrastructure as Code tool to manage your AWS resources, it's possible that your code is set to create a new Task Definition revision on each apply or run.

  3. There might be a misunderstanding in how you're specifying the Task Definition in your Batch job or EventBridge rule. Instead of referencing a specific revision, you should typically reference the Task Definition family without a revision number.

To resolve this issue and use an existing Task Definition revision:

  1. In your AWS Batch job definition or EventBridge rule, make sure you're specifying the Task Definition family name without a revision number. This allows AWS Batch to use the latest active revision automatically.

  2. Review your EventBridge scheduler configuration. Ensure it's not set to create a new Task Definition revision each time it runs.

  3. If you're using Terraform or another IaC tool, check your code to make sure it's not inadvertently creating new revisions.

  4. Consider using the AWS Batch console or CLI to manually deactivate unnecessary Task Definition revisions to clean up your environment.

By making these adjustments, you should be able to prevent the creation of unnecessary Task Definition revisions and have AWS Batch use an existing revision for your nightly process.
Sources
EventBridge scheduler schedules wrong ECS task definition | AWS re:Post
AWS Batch Features

profile picture
re:Post Agent
answered 2 months ago
profile picture
EXPERT
Oleksii Bebych
reviewed 2 months ago
0
Accepted Answer

According to AWS Support the answer is:

"When a new version of the JobDefinition is created, then during the next “SubmitJob” API Call, AWS Batch automatically creates a new version of the associated TaskDefinition to ensure that the task definition remains compatible with the updated job definition. This behavior is by design and is a fundamental aspect of AWS Batch's versioning system."

This kinda makes sense I guess, but it's still a bit surprising. I didn't realize that ECS Tasks and AWS Batch were so closely related. In fact, the only relation that I can find is that the TaskDefinition and JobDefinition share the same container image.

profile picture
RNHurt
answered a month ago
  • RNHurt
    a month ago

    Part of my confusion on this was that my ECS TaskDefinition and Batch JobDefinition resources had the same name (ie. "my-app"). So, every time Batch updated the definition it "overwrote" the TaskDefinition by creating a new one. This was super frustrating and not very clear from the documentation I found.

    One possible solution is simply use different names for the ECS definition (ie. "my-app") and another for the Batch definition (ie "batch-my-app"). At least it keeps ECS from trying to execute "batch" definitions.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content