Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3?

0

Hello, I was wondering if there is a way to automatically name the exported files to S3 with the current date or a specified name when exporting from CloudWatch? Currently, I am looking to send ECS logs to both CloudWatch and S3. My initial approach involves sending ECS logs to CloudWatch and then exporting them to S3. However, I noticed that the exported file is named '0000.gz'. Ideally, I would prefer the file to be named with the current date or month, such as '20240401.gz'. Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3? Alternatively, are there better approaches to achieve the goal of sending ECS logs to both CloudWatch and S3?

1 個回答
1
已接受的答案

Hello.

How about setting up an S3 event notification to notify Lambda when it is exported to S3 and renaming the file in Lambda?
If you can program with Python etc., you can use the Python AWS SDK called boto3.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

It is possible to do something similar to rename by combining "copy_object" and "delete_object".
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_object.html

If you use Kinesis Data Firehose, you can also output files by date hierarchy.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

profile picture
專家
已回答 6 個月前
profile picture
專家
已審閱 6 個月前
profile picture
專家
已審閱 6 個月前
  • Hello Riku, thank you for your response. It seems like a promising approach indeed. However, after some research, I've found that as you mentioned, lambda doesn't directly provide a method to modify file names. Instead, it achieves similar functionality by creating a copy with a new name and then deleting the old one. I'm considering if there might be a simpler solution. If not, I'll certainly consider employing the method you suggested.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南