How do I resolve issues with DynamoDB auto scaling?

Short description

DynamoDB auto scaling creates Amazon CloudWatch alarms on your behalf. When an alarm is set off, the CloudWatch alarm invokes AWS Application Auto Scaling that then notifies DynamoDB to adjust the table's provisioned throughput capacity.

DynamoDB auto scaling modifies provisioned throughput only when the workload stays depressed or elevated for several minutes. For example, you set the minimum read capacity units (RCUs) to 100 and the target utilization to 70 percent:

• DynamoDB auto scaling increases provisioned capacity when utilization exceeds 70 RCUs for at least 2 consecutive minutes.
• DynamoDB auto scaling decreases provisioned capacity when utilization is 20% or more below the target for 15 consecutive minutes (50 RCUs).

Resolution

To troubleshoot problems with DynamoDB auto scaling, complete the following tasks.

Review your CloudWatch alarms

Be sure that you don't delete the CloudWatch alarms that DynamoDB auto scaling creates. When you delete alarms, DynamoDB auto scaling might not work as expected. If you accidentally delete the CloudWatch alarms, then turn off auto scaling and then activate it on the table. CloudWatch automatically recreates the alarms.

Increase your initial throughput capacity

If the capacity is insufficient for the requests made, then the requests are returned with throttling exception that indicates that the requests aren't successful. To solve this issue, increase the initial throughput capacity of the table based on your requirements. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

Use DynamoDB auto scaling for gradual changes in traffic

Don't rely on DynamoDB auto scaling to handle occasional short-duration activity spikes. DynamoDB auto scaling works best when there are gradual increases or decreases in traffic. The table's built-in burst capacity handles occasional activity spikes.

For auto scaling to start the scale up activity, a spike must last for the configured target utilization of minimum write capacity for at least 2 consecutive minutes. For example, if the minimum write capacity configured is 60% of 100 write capacity units, then a use of 60 write capacity units for 2 consecutive minutes starts a scale up activity.

For auto scaling to start a scale down activity, traffic must fall below the target utilization minus 20 percent for 15 consecutive minutes.

Note: CloudWatch is a push based service. Data is pushed from DynamoDB to the CloudWatch metrics. There might be a delay in data being pushed to CloudWatch based on the amount of time it takes DynamoDB to put this information forward.

Use on-demand mode to handle unpredictable workloads

If the table's traffic is frequently unpredictable, then use an UpdateTable operation to set the billing mode to PAY_PER_REQUEST. This update activates on-demand mode to adapt throughput to handle the workload.

Update your scaling activities

Scaling activities are based on the sum per minute of the consumed dimension, not the average statistic. To review your scaling activities, check the Additional settings tab of your table in the DynamoDB console.

You can increase ReadCapacityUnits or WriteCapacityUnits as often as necessary. However, there's a default limit on the number of provisioned capacity decreases that you can perform on your DynamoDB table per UTC day. On a single day, you can perform up to four decreases within one hour as long as you haven't performed any other decreases. After that first hour, you can perform one additional decrease per hour. The maximum number of decreases you can make in a day is 27.

If you make more decreases than what is allowed, then your activities are throttled and the decrease is rejected. You get an error message similar to the following:

"Subscriber limit exceeded: Provisioned throughput decreases are limited within a given UTC day. After the first 4 decreases, each subsequent decrease in the same UTC day can be performed at most once every 3600 seconds. Number of decreases today: 11. Last decrease at Wednesday, December 16, 2020 at 10:04:27 AM Coordinated Universal Time. Next decrease can be made at Wednesday, December 16, 2020 at 11:04:27 AM Coordinated Universal Time (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: LimitExceededException..."

A decrease is allowed up to four times any time per day (GMT time zone). Also, if there is no decrease in the preceding hour, an additional decrease is allowed. The maximum number of decreases in a day is 27 decreases total.

Because auto scaling lowers capacity after consumption, the scale-down period can be slower. This slower period is by design, and follows the auto scaling limits for decreases.

To resolve this issue, lower the amount of increase and decrease scaling activities for your table.

Create a scheduled job for your table

To create a scheduler job for the table to increase and decrease based on your expected traffic, complete the following steps:

1. Create a scheduled scaling action to increase the read capacity units and write capacity units.
2. Create a second scheduled scaling action to decrease the read capacity units and write capacity units to run shortly after the increase action.

Related information

Amazon DynamoDB auto scaling: Performance and cost optimization at any scale