How do I identify the log group that causes a sudden increase in my CloudWatch Logs bill?

4 minute read

My Amazon CloudWatch Logs bill is unusually high. I want to identify the log group that's increasing my CloudWatch Logs costs.

Short description

Sudden increases in the CloudWatch Logs billed amount might result from an increase in ingested or storage data in a particular log group. Use CloudWatch Logs metrics to check data usage. Then, review your AWS bill to identify the log group that's responsible for the increase in your billed amount.

Resolution

Check the amount of data that you're ingesting

The IncomingBytes metric shows you the amount of ingested data in your CloudWatch log groups in near real time. This metric can help you determine the following information:

The log group that's the highest contributor towards your bill.
Any spikes in the incoming data to your log groups or gradual increases because of new applications.
How much data is pushed during a particular period of time.

Query a small set of log groups

Complete the following steps:

Open the CloudWatch console.
In the navigation pane, choose All metrics.
For each of your log groups, complete the following steps:
Choose the Logs namespace.
Choose Log group metrics.
Select the individual IncomingBytes metrics.
Choose the Graphed metrics tab.
For Statistic, choose Sum.
For Period, choose 30 Days.
Choose the Options tab, and then in the Widget type section, choose Number.
At the top of the graph, choose Custom, and then choose Absolute to set the time range. Select a start and end date that corresponds with the last 30 days.

Query a large set of log groups

Complete the following steps:

Open the CloudWatch console.
In the navigation pane, choose All metrics.
Open the Graphed metrics tab. Then, from the Add math dropdown list, select Start with an empty expression.
Copy the following math expression:
SORT(REMOVE_EMPTY(SEARCH('{AWS/Logs,LogGroupName} MetricName="IncomingBytes"', 'Sum', 2592000)),SUM, DESC)
Paste the math expression into the Edit math expression field, and then choose Apply.
Open the Graph options tab, and then choose Number.
At the top of the graph, choose Custom. Then, choose Absolute. Select a start and end date that corresponds with the last 30 days. You can graph up to 500 metrics through this method.
Note: This step only works with the Absolute option. If you choose Relative, then the graph returns incorrect data.
Choose Actions.
Choose Download as .csv.

Query a large set of log groups

Prerequisite: Review the associated costs with making API calls. It's a best practice to distribute the ListMetrics operation to avoid throttling. The default quota for ListMetrics is 25 transactions per second. However, you can request a quota increase.

Complete the following steps:

Run a ListMetrics API operation. Use the following parameters:
**Namespace: AWS/Logs
**MetricName: IncomingBytes
Run a GetMetricData API operation. Use the following parameters:
Namespace: AWS/Logs
MetricName: IncomingBytes
Dimensions: As received from the ListMetrics call
StartTime: Date and time that's 14 days before the current date
EndTime: Current date and time]
Period: EndTime - StartTime, in seconds
Statistics: Sum
To display the log group names with the highest ingested data amounts, sort the data points in descending order.

To be sure that ingested data charges don't exceed a specified quota in the future, create a CloudWatch alarm.

Review your storage data use

The storedBytes parameter represents the number of compressed bytes stored in a log group. To check storedBytes for a log group, complete the following steps:

Open the CloudWatch console.
In the navigation pane, choose Log groups.
Select the log group.
Review the total size of the compressed log events under the Stored bytes parameter.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To retrieve storedBytes for log groups in descending order, run the following describe-logs-groups AWS CLI command:

aws logs describe-log-groups --query 'logGroups | sort_by(@, &storedBytes) | reverse([]) | [].{"LogGroupName": logGroupName, "StoredBytes": storedBytes, "RetentionInDays": retentionInDays}' --output table

Note: If the value in the RetentionInDays column of the AWS CLI output is None, then the NeverExpire setting applies to the respective log group.

The retention periods determine the duration that CloudWatch Logs maintains your data. If you reduce the retention periods, then you can limit the amount of historical data stored over time and reduce archival charges.

It's a best practice to implement retention periods based on scenarios and export historical data to more cost-effective storage solutions, such as Amazon Simple Storage Service (Amazon S3).