How do I avoid throttling when I call PutMetricData in the CloudWatch API?

Short description

The PutMetricData API publishes metric data points to CloudWatch. CloudWatch throttles PutMetricData requests for each AWS account per AWS Region. When you exceed the quota for the rate of PutMetricData requests, CloudWatch returns the "400 ThrottlingException" error with the message "Rate exceeded."

To avoid throttling, reduce your PutMetricData call rate. You can batch metrics into fewer calls, spread out calls over time, or switch to embedded metric format..

Note: All PutMetricData calls in a Region count toward the request rate. This includes calls from any custom or third-party application, such as the CloudWatch Agent, the AWS Command Line Interface (AWS CLI), and the AWS Management Console.

Resolution

Identify the source of throttling

Sources of high PutMetricData call volume include the CloudWatch agent, third-party monitoring agents, custom application code, and multiple AWS Lambda functions that publish metrics at the same time.

Identify the AWS Identity and Access Management (IAM) role, user, or service that makes the most PutMetricData calls.

Complete the following steps:

1. Open the CloudTrail console.
2. In the navigation pane, choose Event history.
3. For Event name, select PutMetricData.
4. For Error code, select Throttling.
5. Choose the Username column, and then identify the source.

Check your PutMetricData call rate

To check how close you are to your quota, complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose All metrics.
3. Choose Usage, and then for the filter, choose By AWS Resource.
4. In the list of metrics, find the row where the Resource dimension is PutMetricData and the Service dimension is CloudWatch.
5. Compare the call count against your quota in the Service Quotas console.
6. Set a CloudWatch alarm at 80% of your quota to receive a notification before throttling starts.

Batch metrics in a single API call

Each PutMetricData call supports up to 1,000 metrics and 150 data points per metric. If your application sends one metric per API call, then combine them into a single call with up to 1,000 metrics.

Use pre-aggregated data sets, such as StatisticSet, to publish aggregated values including Sum, Min, Max, and SampleCount instead of individual data points. When you batch metrics, this reduces the number of PutMetricData calls per second.

Spread out your calls and add retry logic

If your application sends many calls at the same time, for example, every minute on the minute, then add jitter to introduce a random delay.

You can also retry failed PutMetricData calls with exponential backoff and jitter. If you use an AWS SDK, then retry logic is already applied by default. If you use custom HTTP clients, then add this logic manually. For more information, see Timeouts, retries, and backoff with jitter.

Use embedded metric format for high-volume metrics

If your application consistently approaches or exceeds the PutMetricData transactions per second (TPS) quota, or publishes metrics from many concurrent sources such as hundreds of Lambda invocations, then use embedded metric format instead of PutMetricData. With embedded metric format, you write metrics as structured JSON inside your application logs. CloudWatch automatically extracts the metrics from the logs.

Use the CloudWatch agent for automatic batching

To avoid direct PutMetricData calls from your code, use StatsD or collectd to send metrics locally to the CloudWatch agent. The agent automatically batches, aggregates, and retries metric submissions. For more information, see Retrieve custom metrics with StatsD and Retrieve custom metrics with collectd.

Reduce metric cardinality

If you use unique values as dimensions such as request IDs or user IDs, then each unique combination creates a separate metric stream that increases your API call volume. Use broader categories as dimensions, such as environment name or service name, rather than unique identifiers like request IDs or user IDs. For high-cardinality analysis, use CloudWatch Logs Insights.

Related information

CloudWatch API reference

Managing and monitoring API throttling in your workloads