How can I optimize costs with CloudWatch Logs costs for high-volume data ingestion and long-term storage?

Resolution

Identify high-volume log groups

Use CloudWatch metric math expressions to identify log groups with high data ingestion:

SORT(SEARCH('{AWS/Logs,LogGroupName} MetricName="IncomingBytes"', 'Sum', 86400), MAX, 10)

This expression helps you identify the top 10 log groups by incoming data volume over the last 24 hours.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Implement log retention policies

Review and modify retention periods for your log groups through these steps:

1. Open the CloudWatch console.
2. Navigate to the Log groups section.
3. Select the log group that you want to modify.
4. From the Actions menu, choose Edit retention setting.
5. Set an appropriate retention period (for example, 7, 14, or 30 days) based on your compliance and operational requirements.

Use the Infrequent Access log class

For log groups that you don't frequently access, create new log groups with the Infrequent Access class. For more information, see CreateLogGroup.

Note: You can only assign this log group class during creation. You can't change it afterward.

Reduce log verbosity

Review your applications and AWS services to reduce unnecessary logging through the following actions:

• Analyze your log data to identify and remove debug-level or repetitive messages. For more information, see Analyzing log data with CloudWatch Logs Insights.
• For AWS Lambda functions, you can modify the execution role policy to limit log ingestion if appropriate for your use case.
• For AWS Glue jobs, implement strategies to reduce CloudWatch Logs output.

Implement log filtering and aggregation

To implement log filtering and aggregation, take the following actions:

• Use CloudWatch Logs Insights to analyze log patterns and identify opportunities for filtering.
• Implement CloudWatch Logs metric filters to create custom metrics based on specific log patterns.
• Use AWS Lambda functions to process and aggregate logs before ingestion. For more information, see Create your first Lambda function.

Use Amazon S3 for long-term storage

To use Amazon Simple Storage Service (Amazon S3) for long-term storage, take the following actions:

• Set up a CloudWatch Logs subscription filter to send logs to Amazon S3 for archival. For more information, see Export log data to Amazon S3 using the console.
• Implement a log export process to move older logs from CloudWatch to Amazon S3.
• Use Amazon S3 Lifecycle policies to transition logs to cheaper storage tiers (for example, Amazon S3 Glacier) for long-term retention.

Monitor and analyze costs

Monitor and analyze costs through the following actions:

• Use AWS Cost Explorer to track CloudWatch Logs costs over time. For more information, see Getting started with Cost Explorer.
• Set up budget alerts with AWS Budgets to alert you when CloudWatch Logs costs exceed predefined thresholds.
• Regularly review and audit unused CloudWatch metrics and alarms.

Consider a centralized logging solution

For multi-Region and multi-account setups, evaluate centralized logging solutions through services such as Amazon OpenSearch Service or third-party log management tools. This can help consolidate logs from multiple sources and potentially reduce overall costs.

Related information

Amazon CloudWatch pricing

What is Amazon CloudWatch Logs?