How do I resolve log ingestion issues with the CloudWatch agent and reduce costs?

Short description

If you didn't correctly configure the CloudWatch agent, then log ingestion fails and CloudWatch Logs doesn't display your logs.

CloudWatch logs might not display in the following situations:

• The CloudWatch agent doesn't get and apply the configuration file.
• Your Amazon Elastic Compute Cloud (Amazon EC2) instances don't have the required AWS Identity and Access Management (IAM) permissions.
• The EC2 instance can't connect to the CloudWatch Logs endpoint.
• The agent missed logs.
• The monitored log file lacks a trailing newline character.

To reduce log ingestion costs, use log filters to send only the logs that your application requires to CloudWatch Logs.

Resolution

Review CloudWatch agent logs for errors

To identify configuration or permissions issues, review the log files that the CloudWatch agent generates.

You can find the log files in the following paths:

• Linux: /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
• Windows: $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log or C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

To view debug logs, configure the agent to log additional details in the amazon-cloudwatch-agent.log file.

In the agent section of the configuration file, set the debug field to true:

"agent": {"debug": true,}

After you modify the configuration file, run the following fetch-config command to restart the CloudWatch agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:CONFIGURATION-FILE-PATH

Note: Replace CONFIGURATION-FILE-PATH with the full path to your configuration file.

For more information, see Troubleshooting the CloudWatch agent.

The CloudWatch agent doesn't get and apply the configuration file

To resolve this issue, complete the following steps:

1. Run the following command to stop the CloudWatch agent:

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop -m ec2

2. Run the following fetch-config command to get and apply the configuration file:

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:CONFIGURATION-FILE-PATH

   Note: Replace CONFIGURATION-FILE-PATH with the full path to your configuration file. If you used the wizard to create the configuration file, then the path is /opt/aws/amazon-cloudwatch-agent/bin/config.json.

3. Confirm that the log group appears in CloudWatch Logs and that CloudWatch receives log data.
   Note: If you start the agent without the fetch-config command, then the agent runs with its existing configuration and doesn't apply your updated configuration.

Your EC2 instances don't have the required IAM permissions

To use the agent with instances, you must create an IAM role and attach the CloudWatchAgentServerPolicy policy to it. Then, attach the IAM role to the instances.

The instance can't connect to the CloudWatch Logs endpoint

To test endpoint connectivity, run one of the following commands.

Telnet:

telnet logs.REGION.amazonaws.com 443

Netcat:

nc -zv logs.REGION.amazonaws.com 443

Note: Replace REGION with your AWS Region.

If the connection fails, then take the following actions:

• Check that the security group and network access control list (network ACL) rules allow connectivity on port 443.
• Check that the instance reaches the public endpoint through an internet gateway or NAT gateway.
• If you use virtual private cloud (VPC) endpoints, then confirm that the endpoint uses a VPC IP address instead of a public IP address. Also, check that the endpoint security group allows access from the source instance.

The agent missed logs

The CloudWatch agent sends logs only from the most recently modified file when both of the following conditions are true:

• You use wildcards in the log file path.
• Multiple files in different directories change within a short period.

The agent doesn't send logs from all other matching files to CloudWatch Logs. For example, your configuration file specifies the path logs\LogFiles***.log and you have folder1 and folder2 inside the LogFiles directory. The agent might not collect logs from both folders when you write new log events to both folders within a short period.

To resolve this issue, explicitly define each log folder in the CloudWatch agent configuration file.

For example, if you have folder1 and folder2 inside the LogFiles directory, then specify each folder path separately:

logs\LogFiles\folder1*.log

logs\LogFiles\folder2*.log

The monitored log file lacks a trailing newline

The CloudWatch agent uses a newline character as the delimiter for a single log event. If a monitored log file doesn't end with a newline character, then the agent doesn't send any log events from that file. The agent doesn't output an error or warning message for this condition.

This behavior occurs in the following scenarios:

• An application writes log entries without appending a newline at the end of the file.
• A process overwrites the log file with binary data that contains no newline characters.

To resolve this issue, complete the following steps:

1. Confirm that your application appends a newline character (\n) at the end of each log entry.

2. Run the following command to check whether the monitored file ends with a newline:

   tail -c 1 /path/to/your/logfile | xxd

3. If the output doesn't show 0a (the hex value for a newline), then the file lacks a trailing newline. Update your application to append a newline after each log entry.

Note: If the agent logs an INFO-level message "Re-opening truncated file," then the agent detected that the monitored file's size decreased. This message indicates file truncation (not a newline issue) and the agent re-reads the file from the beginning.

Reduce log ingestion costs

Log filters use regular expressions to include or exclude specific log events before the agent sends them to CloudWatch Logs. The agent drops log events that don't match the filter criteria. For more information, see Optimize log collection with Amazon CloudWatch agent log filter expressions.

To add log filters, add a filters section to each log stream in the collect_list section of the agent configuration file:

"collect_list": [ {"file_path": "/var/log/messages", "log_group_name": "syslogs", "log_stream_name": "{instance_id}","filters": [{"type": "include","expression": "systemd:"},{"type": "exclude","expression": "Message Of The Day"}]}]

In the preceding configuration, the CloudWatch agent sends only log lines that contain the systemd: string. The agent excludes log lines that contain the Message Of The Day string.

Note: The agent evaluates filters in the order that they appear, and the order affects the performance of the agent. In the preceding example, the configuration improves agent performance because the agent drops all logs that don't contain systemd: before it evaluates the exclude filter.

After you modify the configuration file, run the fetch-config command to restart the CloudWatch agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:CONFIGURATION-FILE-PATH

Note: Replace CONFIGURATION-FILE-PATH with the full path to your configuration file.

Related information

Installing the CloudWatch agent

CloudWatch agent files and locations