How do I pause a queue and cancel thousands of jobs in that queue?

Short Description

Queues can be ACTIVE or PAUSED. If you pause a queue, jobs in that queue won’t begin. If you pause a queue, the service won't begin processing jobs in that queue. You can cancel a job using either the MediaConvert console or CLI. However, Cancelling thousands of tasks using console is not only time-consuming but also annoying. Therefore, we introduce you to cancel with a simple shell script using the CLI.

The idea is :

• Pause a queue
• Retrieve list of jobs in a queue
• Cancel a job
• Iterate over a list using loop in bash

Steps

1. Pause on-demand queues

To change the status of an on-demand queue by using the MediaConvert console:

• Open the Queues page in the MediaConvert console.
• In the On-demand queues section, select the queue.
• Choose Edit queue.
• Under Status, choose Pause or Active.
• Choose Save queue.

2. Retrieve list of jobs in a queue

• Take a note the mediaconvert endpoint of the region where you want to cancel the MediaConvert job (ex. ap-northeast-2)

• The endpoint for ap-northeast-2 is “https://bnklbqvoa.mediaconvert.ap-northeast-2.amazonaws.com”.

$ aws mediaconvert describe-endpoints --region ap-northeast-2
{
    "Endpoints": [
        {
            "Url": "https://bnklbqvoa.mediaconvert.ap-northeast-2.amazonaws.com"
        }
    ]
}

• Use the list-jobs [2] command in MediaConvert to list the job IDs in the SUBMITED state.

aws mediaconvert list-jobs --region ap-northeast-2 --endpoint-url https://bnklbqvoa.mediaconvert.ap-northeast-2.amazonaws.com --status SUBMITTED --queue Default --query 'Jobs[*].{Id:Id}' --output text

3.Cancel a job[3] using cancel-job in MediaConvert

The following cancel-job example cancels a job.

aws mediaconvert cancel-job \
--endpoint-url https://abcd1234.mediaconvert.region-name-1.amazonaws.com \
--id 1234567890123-efg456

4. Iterate list of jobs in a queue using bash script

• Use the list-jobs command to determine how many tasks you need to cancel.
• Then write code to cancel jobs using the bash script in units of 50 or 100.

Sample code : Ex) This is the sample code written by MAC BOOK that I am using. You must modify it to suit your environment. (<endpoint url> must include the endpoint output from #1, <regioin, ex. us-east-1> is the same area information as ap-nortaste-2.)

#!/bin/sh
$aws mediaconvert list-jobs --region <regioin, ex. us-east-1> --endpoint-url <endpoint url> --status SUBMITTED --queue Default --query 'Jobs[*].{Id:Id}' --output text | while read id;
do
    echo "Read Job id : $id"
        aws mediaconvert cancel-job --region <regioin, ex. us-east-1> --endpoint-url <endpoint url> --id $id > /dev/null
    echo "Cancel Job id : $id"
done

Additional comments :

Apart from job canceling, I would like to encourage you to have multiple on-demand queues in order to run jobs for different workflows in separate queues.

References :

[1] https://docs.aws.amazon.com/mediaconvert/latest/ug/working-with-on-demand-queues.html#updating-queue-status

[2] https://docs.aws.amazon.com/cli/latest/reference/mediaconvert/list-jobs.html

[3] https://docs.aws.amazon.com/cli/latest/reference/mediaconvert/cancel-job.html