How do I prevent "Rate exceeded" errors in CloudFormation?

Short description

API calls to an AWS service that exceed the maximum allowed API requests generate the Rate exceeded error. When this occurs, API calls are throttled.

Resolution

A downstream AWS service exceeds the maximum allowed rate

To prevent an AWS service from exceeding the maximum allowed rate during CloudFormation stack operation, use the following options.

Use the DependsOn attribute

Unless a dependency is defined between resources, CloudFormation creates and updates resources at the same time. The DependsOn attribute defines dependencies between resources to control concurrent updates.

To specify when each dependent resource is created or updated, use the DependsOn attribute. For example, if resource B is dependent on resource A, then you can specify that resource A must be created or updated before resource B. This specification limits the number of API calls that happen at the same time and reduces the occurrence of throttling. You can also use the DependsOn attribute with nested stacks.

Request a quota increase

If the DependsOn attribute doesn't resolve the Rate exceeded error, then you can request a quota increase. Before you request a quota increase for the downstream API, first identify the API call to determine the one that's exceeding the call rate.

In your request for a quota increase, include your AWS Region, the time frame of the API throttling, and the reason for the increase.

An AWS CloudFormation API exceeds the maximum allowed rate

To prevent the CloudFormation API calls from exceeding the maximum allowed API requests, use the following options.

Implement exponential back off

When you use AWS API endpoints, implement back off to decrease the number of API calls.

The following example pseudocode for a DescribeStacks API call is configured to retry the API call after a specific time frame:

{
Make 'DescribeStacks' API call

if throttled: wait 2 sec; Make 'DescribeStacks' API call 
if throttled: wait 4 sec; Make 'DescribeStacks'API call 
if throttled: wait 8 sec; Make 'DescribeStacks' API call 
if throttled: wait 16 sec; Make 'DescribeStacks' API call 
if throttled: wait 32 sec; Make 'DescribeStacks' API call 
}

Create or update stacks one at a time

Multiple CloudFormation stacks created or updated at the same time can cause too many API calls. To prevent the API calls from exceeding the maximum allowed API requests, create or update one stack at a time.

Request a quota increase

If you can't create or update stacks, then you can request a quota increase for the CloudFormation API. Before you request a quota increase, first identify the API call to determine the one that's exceeding the call rate.

In your request for a quota increase, include your AWS Region, the time frame of the API throttling, and the reason for the increase.