How do I troubleshoot throttling errors in my CloudWatch logs?

Short description

When you work with CloudWatch logs, you might exceed the API rate quota. When this happens, your API call is throttled, and you receive a RequestLimitExceeded or ThrottlingException error. To resolve these errors, identify where and when throttling errors happen. Then, make informed rate quota increase requests.

Resolution

Intermittent throttling errors on CloudWatch Logs when you access logs

To list your log events or log streams, use the FilterLogEvents or GetLogEvents API calls. These API calls don't qualify for a quota increase. However, when you use the FilterLogEvents API to search for log events from a specified log group, there is a quota. The quota is five transactions per second (TPS) per AWS account or AWS Region. If you reach this quota, then you receive the RateExceeded error.

To avoid these throttling errors, use a subscription filter to immediately retrieve log data from CloudWatch logs in real time. For more information, see Log group-level subscription filters and Real-time processing of log data with subscriptions.

It's also a best practice to use CloudWatch Logs Insights to quickly get log data from CloudWatch logs. To view specific log groups, use queries to filter your logs. Make sure to export log data to Amazon Simple Storage Service (Amazon S3) for batch use cases.

Note: Log data can take up to 12 hours to process before you can export the data from CloudWatch Logs. It's not a best practice to use this method for real-time analysis and processing.

ThrottlingException errors when you use a script to retrieve CloudWatch log data

To collect CloudWatch logs, you can develop a collector script. To pull data from different log streams or different time frames in the same log group, create the script that makes a DescribeLogStreams or GetLogEvents API call. However, API calls such as FilterLogEvents, GetLogEvents and DescribeLogStreams are designed for human interaction and not for automation. If you use these APIs in your script, the API call is throttled, and you receive an error.

To avoid throttling errors in this use case, use exponential backoff and retries when you make an API call. For more information, see Exponential backoff and jitter and Retry behavior. Make sure to distribute your API calls over time. Schedule actions with randomization so that they are spread over a period of time.

You can cause rate error if you send API calls in rapid succession. To avoid this issue, add sleep intervals between consecutive API calls. Also, add a delay between API calls that the same script or application sends.

When you use a security information and event management (SIEM) solution such as Splunk to retrieve logs from CloudWatch, the Splunk plugin can cause API throttling errors. To avoid this issue, create a CloudWatch logs subscription filter with Amazon Data Firehose. Then, deliver the log data to Splunk. For more information, see Introduction to the Splunk Add-on for Amazon Web Services on the Splunk website.

Throttling errors when you integrate PutLogEvents API calls with a Lambda function

When you integrate the PutLogEvents API call with your AWS Lambda function, PutLogEvents uploads logs to a specified log stream in batches of 1 MB. This API has a rate quota of 5,000 transactions per second, per account, per Region.

For more information and to request a quota increase, see CloudWatch Logs quotas.

Manage your CloudWatch Logs service quotas

AWS defines quotas for services to protect performance and to be sure of availability. CloudWatch has quotas for metrics, alarms, API request, and alarm email notifications. To see your service quotas and set alarms if you reach the threshold, complete the following steps:

1. Open the Service Quotas console.
2. In the navigation pane, choose AWS services.
3. From the AWS services list, search for Amazon CloudWatch logs.
4. The Service quotas list shows you several attributes or options: the service quota name, the applied value (if it's available), AWS default quota, and whether the quota value is adjustable.
5. To see more information about a service quota, choose the quota name. For example, if you choose GetLogEvents throttle limit in transactions per second, then you see the following items:
   Description
   Quota code
   Quota ARN
   Utilization: %
   Applied quota value
   AWS default quota value
   Adjustable: Y/N
6. In the Amazon CloudWatch alarms section, choose Create alarm, and enter an Alarm threshold and Alarm name.

Related information

How do I avoid throttling when I call PutMetricData in the CloudWatch API?