How do I troubleshoot CloudWatch logs that fail to export to S3 buckets?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you use the most recent AWS CLI version.

Troubleshoot your logs that fail to export to Amazon S3 buckets based on whether they fail when or after you create the export task.

Within export task creation

To troubleshoot tasks that fail when you create them, take the following actions:

• AWS Region: Confirm that your CloudWatch log streams and Amazon S3 buckets are in the same Region.
• Amazon S3 bucket policies: By default, Amazon S3 buckets and objects are private. The resource owner has access only to the bucket and to the objects in the bucket. For more information, see Step 3: Set permissions on an S3 bucket.
• Amazon S3 bucket prefixes: When you set the Amazon S3 bucket policy, it's a best practice to include a randomly generated string as the prefix for the bucket. If you use a prefix, then you must specify the generated string in the Amazon S3 bucket prefix settings when you create the export task. If you don't specify the generated string, then the export task creation fails.
• AWS Identity and Access Management (IAM) policies: Confirm that the AWS Identity and Access Management (IAM) user or role that created the export task has access to Amazon S3 and CloudWatch Logs. For more information, see Step 2: Set up access permissions.
• Check the CloudTrail logs: Search for the CreateExportTask event in AWS CloudTrail to identify the error message. CreateExportTask is an asynchronous operation. If you provide all the necessary information, then an export task is initiated and returns the task's ID. When the task begins, use DescribeExportTasks to check the status. For more information, see Errors.
• Resource quotas: The CloudWatch Logs service quota allows only one running or pending export task per AWS account per Region. You can't change this quota. Make sure that you're in the allowed quota.
• Type of server side encryption: Make sure that you use a supported server-side encryption for your Amazon S3 buckets. Supported encryptions include Advanced Encryption Standard (AES)-256 and server-side encryption that has AWS Key Management Service (AWS KMS) keys. These supported encryptions are known as SSE-KMS.

After export task creation

To troubleshoot tasks that fail after you create them, check the time range setting. If you export log streams that have large amounts of data and specify a long time range, then the export task might fail. To resolve this issue, configure shorter time ranges to export log streams that have lower data amounts.

To check the status of the time range setting, run the following AWS CLI describe-export-tasks command:

aws logs describe-export-tasks --task-id example-task-id

Note: Enter your task ID in place of example-task-id.

To export data to Amazon S3, run the following AWS CLI create-export-task command:

aws logs create-export-task --task-name "example-task-name" --log-group-name "/aws/lambda/example-log-group-name" --destination "example-bucket-name --from 1710378735 --to 1710465135

Note: Enter your task name in place of example-task-name. Enter your log group name in place of example-log-group-name. Enter your destination bucket in place of example-bucket-name.

To check the status of the export task, run the following AWS CLI describe-export-tasks command:

aws logs describe-export-tasks --task-id example-task-id

Note: Enter your task ID in place of example-task-id.

The logs might take up to 12 hours to be available for export. The export task can also take up to a few hours to process. This process depends on the size of the data that you export. It's a best practice to use log group-level subscription filters to process in real time or to continuously archive new data to Amazon S3. For more information, see Real-time processing of log data with subscriptions.

Related information

I configured Amazon CloudWatch to export log data to Amazon S3, but the log data is either missing or invalid. How do I resolve this issue?

How do I retrieve log data from CloudWatch Logs?

Exporting log data to Amazon S3