Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3?

0

Hello, I was wondering if there is a way to automatically name the exported files to S3 with the current date or a specified name when exporting from CloudWatch? Currently, I am looking to send ECS logs to both CloudWatch and S3. My initial approach involves sending ECS logs to CloudWatch and then exporting them to S3. However, I noticed that the exported file is named '0000.gz'. Ideally, I would prefer the file to be named with the current date or month, such as '20240401.gz'. Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3? Alternatively, are there better approaches to achieve the goal of sending ECS logs to both CloudWatch and S3?

1개 답변
1
수락된 답변

Hello.

How about setting up an S3 event notification to notify Lambda when it is exported to S3 and renaming the file in Lambda?
If you can program with Python etc., you can use the Python AWS SDK called boto3.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

It is possible to do something similar to rename by combining "copy_object" and "delete_object".
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_object.html

If you use Kinesis Data Firehose, you can also output files by date hierarchy.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

profile picture
전문가
답변함 6달 전
profile picture
전문가
검토됨 6달 전
profile picture
전문가
검토됨 6달 전
  • Hello Riku, thank you for your response. It seems like a promising approach indeed. However, after some research, I've found that as you mentioned, lambda doesn't directly provide a method to modify file names. Instead, it achieves similar functionality by creating a copy with a new name and then deleting the old one. I'm considering if there might be a simpler solution. If not, I'll certainly consider employing the method you suggested.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠