- 最新
- 投票最多
- 评论最多
Hello,
The error you are getting for your job “test” with job run ID “jr_6eb6af04d2a560f71d935ab3fca35504d7fdb99b748c0e0266e71402ced4437f“ is indeed due to the throttling of the resources that are allocated for your account.
You can refer the below document for more detail on the default quota limits: https://docs.aws.amazon.com/general/latest/gr/glue.html
Please find below the responses for your questions:
Q1) Is this error, because the glue jobs are maybe still running in the background ?
Your assumption is correct. In case there are glue jobs running, the resources are allocated for that job and if you try to run another job in parallel and if there are not enough resources to satisfy the requirements the job will fail. This is due to the fact that the quota limits are set for the whole account.
Please refer to the below steps to increase the service quota limits from your console:
- Open Service Quotas console in AWS
- Click on AWS services from the left pane and search for Glue
- Click on AWS Glue and search for Quota name you want to increase
- Click on respective service quota and select "Request quota increase"
- Add new value under "Change quota value" which will be auto approved.
Q2) Is there a way to list and kill these still running glue jobs to free up these resources? I have already tried with awscli aws glue batch-get-jobs --job-names ..., but no joy here of listing them.
To view the metadata for all runs of given job you can use “get-job-runs”. Usage: aws glue get-job-runs --job-name “test”
Please refer to https://docs.aws.amazon.com/cli/latest/reference/glue/get-job-runs.html for more details and usage.
To stop one or more job runs for a specified job you can use the “batch-stop-job-run” cli command. Usage: aws glue batch-stop-job-run --job-name “test” --job-run-ids “jrxxxxxxx”
Please refer to https://docs.aws.amazon.com/cli/latest/reference/glue/batch-stop-job-run.html for more details and usage.
相关内容
- AWS 官方已更新 3 年前
- AWS 官方已更新 2 年前
- AWS 官方已更新 3 年前
- AWS 官方已更新 2 年前
I managed to get this fix by setting the "Max task dpus per account" back to the default 500, since it has somehow been set to 0 ( not by me, never seen this quota page before, maybe due to a similar reason as https://repost.aws/questions/QU1G873wSmT9GDAicnP9RI3Q/aws-glue-task-allocated-capacity-exceeded-limit-solved-but-error-persist-at-notebook-session-creation ) and also up'ing my "Max jobs per account" too.
I hope this helps future developers to get out of this sticky situation with glue :-)