How do I prevent ThrottlingException or Rate exceeded errors when using AWS Glue?

2 minute read

When I make an AWS Glue API call, I receive a ThrottlingException error message indicating that my rate is exceeded.

Short description

When you make an AWS Glue API request, you might receive an error similar to one of the following:

"Error Code: ThrottlingException; Request ID: 07512ede-123f-49c8-8933-1298742297b9; Proxy: null)" } }{ "taskSubmitFailedEventDetails": { "resourceType": "glue", "resource": "startJobRun.sync", "error": "Glue.AWSGlueException", "cause": "Rate exceeded (Service: AWSGlue; Status Code: 400;"
"TaskSubmitFailed": {"resourceType": "glue", "resource": "startJobRun.sync", "error": "Glue.AWSGlueException", "cause": "Rate exceeded (Service: AWSGlue; Status Code: 400; Error Code: ThrottlingException; Request ID: 70ea5ab7-2bb8-4a47-b957-799ef46c129e; Proxy: null)" }"
"botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the GetCrawler operation (reached max retries: 4): Rate exceeded cause": "Rate exceeded (Service: AWSGlue; Status Code: 400; Error Code: ThrottlingException; Request ID: 70ea5ab7-2bb8-4a47-b957-799ef46c129e; Proxy: null)"

AWS Glue API requests are throttled for AWS accounts on a per-Region basis to help service performance. For more information, see AWS Glue endpoints and quotas.

Follow these best practices to mitigate a ThrottlingException or rate exceeded error:

Reduce the frequency of the API calls.
Stagger the intervals between your API calls so that they don't all run at the same time.
Use APIs that return more than one value. For example, GetPartitions supports a call of 1000 values with one API call. Similarly, GetTables supports a call of 100 parameters with one API call.
Implement error retries and exponential backoff when you make API calls.
Using the AWS CloudTrail console, check which and how many API calls are sent during a given period of time.

If you've tried these troubleshooting steps but still receive rate exceeded errors, then request a service quota increase.

Note: Before submitting a request to increase your quota, identify the API call that caused the error and the call rate.

Topics