Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3?

0

Hello, I was wondering if there is a way to automatically name the exported files to S3 with the current date or a specified name when exporting from CloudWatch? Currently, I am looking to send ECS logs to both CloudWatch and S3. My initial approach involves sending ECS logs to CloudWatch and then exporting them to S3. However, I noticed that the exported file is named '0000.gz'. Ideally, I would prefer the file to be named with the current date or month, such as '20240401.gz'. Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3? Alternatively, are there better approaches to achieve the goal of sending ECS logs to both CloudWatch and S3?

1回答
1
承認された回答

Hello.

How about setting up an S3 event notification to notify Lambda when it is exported to S3 and renaming the file in Lambda?
If you can program with Python etc., you can use the Python AWS SDK called boto3.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

It is possible to do something similar to rename by combining "copy_object" and "delete_object".
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_object.html

If you use Kinesis Data Firehose, you can also output files by date hierarchy.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

profile picture
エキスパート
回答済み 6ヶ月前
profile picture
エキスパート
レビュー済み 6ヶ月前
profile picture
エキスパート
レビュー済み 6ヶ月前
  • Hello Riku, thank you for your response. It seems like a promising approach indeed. However, after some research, I've found that as you mentioned, lambda doesn't directly provide a method to modify file names. Instead, it achieves similar functionality by creating a copy with a new name and then deleting the old one. I'm considering if there might be a simpler solution. If not, I'll certainly consider employing the method you suggested.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ