How do I retrieve log data from CloudWatch Logs?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To retrieve your log data from CloudWatch Logs, use the following best practices.

Stream log data with subscription filters

To retrieve your log data from CloudWatch Logs in real time, use subscription filters. Log data subscription filters prevent CloudWatch API throttling. For more information, see Real-time processing of log data with subscriptions and Log group-level subscription filters.

Note: Amazon Kinesis Data Streams automatically retries throttled service APIs.

Use CloudWatch APIs

To manually locate your log data, use the GetLogEvents or FilterLogEvents CloudWatch API operations. You can't scale these API operations because of the transactions per second (TPS) quota. The GetLogEvents API operation might return empty results when there are more log events available through the token. For current quotas, see CloudWatch Logs quotas. If you experience API throttling, then use subscription filters.

Run a CloudWatch Logs Insights query

To search and analyze your log data, run a query in CloudWatch Logs Insights. Queries help you respond to issues and identify causes for further troubleshooting. For more information, see Analyzing log data with CloudWatch Logs Insights.

Export log data to Amazon S3

You can export CloudWatch Logs data to Amazon S3 in batch use cases. The log data export can take up to 12 hours to become available from CloudWatch Logs. For real-time analysis and processing, use subscription filters.

Note: An AWS account can have only one export task for log data in the PENDING, PENDING_CANCEL, or RUNNING state.

Download logs into a .csv file

You can use the CloudWatch console, AWS CLI, or CloudWatch Logs Insights to retrieve your log data.Use the CloudWatch console

Complete the following steps:

1. Open the CloudWatch console.
2. Select your log group.
3. Select the log stream that contains the log events that you want to download.
4. Choose Actions, and then choose Download Search Results.

Use AWS CLI

To retrieve log events in .csv file format, run the following get-log-events AWS CLI command:

aws logs get-log-events --log-group-name example-loggroup-name  --log-stream-name example-logstream-name  --start-time 1705147529 --end-time 1705233929 log_events.csv

Note: Replace example-loggroup-name with your log group name, example-logstream-name with your log stream name, 1705147529 with your start time and 1705233929 with your end time.

The .csv file is stored in the same directory that the command was initiated from. Also, the AWS CLI uses the default AWS Region and profile that's configured for the workstation. Specify the Region for your workstation in the preceding AWS CLI command.

Use CloudWatch Logs Insights to retrieve your log data

Run a query to retrieve your log data. To export data with fewer than 10,000 logs, choose Export results. Or, to export data for more than 10,000 logs, export the logs to an S3 bucket, and then download the data into a .csv file.