How do I troubleshoot throttling errors for CloudWatch Logs API calls?

Short description

The "RequestLimitExceeded" and "ThrottlingException" errors occur when your API calls exceed the CloudWatch Logs rate quota. To resolve these errors, identify where and when throttling occurs. Then, request a rate quota increase based on your findings.

Resolution

Resolve intermittent throttling errors when you access logs

If you use the FilterLogEvents or GetLogEvents API calls to list log events or log streams, then your API calls might be throttled. These API calls don't qualify for a quota increase. When you use the FilterLogEvents API to search for log events from a specified log group, the quota is five transactions per second (TPS) per AWS account or AWS Region. If you exceed this quota, then you receive the "RateExceeded" error.

To avoid the "RateExceeded" error, use a subscription filter to immediately retrieve log data from CloudWatch logs in real time. For more information, see Log group-level subscription filters and Real-time processing of log data with subscriptions.

It's a best practice to use CloudWatch Logs Insights to quickly retrieve log data. To view specific log groups, use queries to filter your logs. For batch use cases, export log data to Amazon Simple Storage Service (Amazon S3).

Note: CloudWatch Logs takes up to 12 hours to process log data before you can export the data. Don't use this method for real-time analysis and processing.

Resolve "ThrottlingException" errors when you use a script to retrieve log data

If you use FilterLogEvents, GetLogEvents, or DescribeLogStreams API calls in a collector script to retrieve data from different log streams or time frames, then your API calls might be throttled. These API calls are designed for human interaction, not automation. If you use these APIs in your script, then you might receive the "ThrottlingException" error.

To avoid throttling errors, take the following actions.

Use exponential backoff and retries when an API call fails to avoid repeated throttling. For more information, see Timeouts, retries, and backoff with jitter and Retry behavior.

Use randomization to distribute your API calls over time rather than send them in rapid succession. Also, add a delay between consecutive API calls in the same script or application.

If you use a security information and event management (SIEM) solution such as Splunk to retrieve logs from CloudWatch, then the Splunk plugin can cause API throttling errors. To avoid this issue, create a CloudWatch logs subscription filter with Amazon Data Firehose. Then, deliver the log data to Splunk. For more information, see About the Splunk add-on for Amazon Web Services on the Splunk website.

Resolve throttling errors when you integrate PutLogEvents API calls with a Lambda function

When you integrate PutLogEvents with an AWS Lambda function, the PutLogEvents API uploads logs to a specified log stream in batches of up to 1 MB. The PutLogEvents API has a rate quota of 5,000 transactions per second (TPS) per account per Region. If your Lambda function exceeds this quota, then your API calls are throttled.

To resolve this issue, request a quota increase.

Set alarms for CloudWatch Logs service quotas

AWS defines service quotas to protect performance and maintain availability. To view your CloudWatch Logs service quotas and set alarms when you reach the threshold, complete the following steps:

1. Open the Service Quotas console.
2. In the navigation pane, choose AWS services.
3. From the AWS services list, search for Amazon CloudWatch logs.
4. Choose a quota name to view details.
   Note: You can view details such as the Amazon Resource Name (ARN), utilization percentage, applied quota value, default quota value, and whether the quota is adjustable. For example, choose GetLogEvents throttle limit in transactions per second to view its details.
5. In the Amazon CloudWatch alarms section, choose Create alarm, and then enter a threshold value and a name for the alarm.

Related information

How do I avoid throttling when I call PutMetricData in the CloudWatch API?