Cloud Watch agent stopped sending logs after clearing log file

0

I have a working Cloudwatch agent on my EC2 (belonging to an ASG+ALB), streaming my log file (the java tomcat log file catalina.out) to my Cloudwatch log group. I would like to regularly clear my log files on the EC2 to avoid getting a full disk.

When I manually make a log file clear (echo "" > catalina.out), the agent continues to send incoming logs from this file to the log group. But when I'm asking crontab to do it for me, the Cloudwatch agent stops sending the logs. Why? The same user ran the two operations and my file permissions didn't change. This test was done with the old Cloudwatch agent running with Python. I've tried with the latest Cloudwatch agent (amazon-cloudwatch-agent) and, in this case, the clear (echo "" > catalina.out) is enough to break the log streaming. I've tried to stop the Cloudwatch agent before clearing and restarting after, but the issue remains.

How to find out what's going on and where the issue could be?

已提問 1 個月前檢視次數 442 次
1 個回答
2
已接受的答案

Hi THere

Instead of clearing the log file with the echo command, id recommend implementing something like logrotate or one of the supported methods below:

From https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html#agent-faq

The following file rotation mechanisms are supported:

  1. Renaming existing log files with a numerical suffix, then re-creating the original empty log file. For example, /var/log/syslog.log is renamed /var/log/syslog.log.1. If /var/log/syslog.log.1 already exists from a previous rotation, it is renamed /var/log/syslog.log.2. (logrotate will do this for you)
  2. Truncating the original log file in place after creating a copy. For example, /var/log/syslog.log is copied to /var/log/syslog.log.1 and /var/log/syslog.log is truncated. There might be data loss for this case, so be careful about using this file rotation mechanism.
  3. Creating a new file with a common pattern as the old one. For example, /var/log/syslog.log.2014-01-01 remains and /var/log/syslog.log.2014-01-02 is created.

You might also find the exact reason that the logs stop streaming by examining the CloudWatch agent log file. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html#CloudWatch-Agent-troubleshooting-loginfo But I would still try and stick with one of the supported rotation mechanisms to ensure you do not lose data.

profile pictureAWS
專家
Matt-B
已回答 1 個月前
profile picture
專家
已審閱 1 個月前
  • Thanks for your help. The solution was to use logrotate instead of directly writing a cron. Delete or move the catalina.out file wasn't possible as would break the tomcat logs, even if we created a new empty file after. That's why I used cron to follow the 2nd point you've raised here. Don't know why logrotate made it work and not cron, but at least it's working!

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南