Why doesn't the unified CloudWatch agent push my metrics or log events to CloudWatch?

Short description

The unified CloudWatch agent might not push metrics or logs to CloudWatch if there are connectivity problems or permission issues. When you review the unified CloudWatch agent logs, one of the following errors might appear:

• Agent Log Error: No Connectivity to Endpoint
• Agent Log Error: Insufficient Permissions

Resolution

To troubleshoot the unified CloudWatch agent, complete the following steps:

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version..

Review the unified CloudWatch agent logs

You can use the agent log file to help troubleshoot problems that you encounter with the unified CloudWatch agent package.

You might experience one the following issues:

• You can’t connect to required AWS service endpoints or Amazon Virtual Private Cloud (Amazon VPC) endpoints. For more information, see Work with VPC endpoints.
• You don't have the correct permissions to make support API calls to CloudWatch. For more information, see CloudWatch API operations and required permissions for actions.

You might see one of these errors in the following logs.

Agent Log Error: No Connectivity to Endpoint

2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 210 retries, going to sleep 1m0s before retrying.
2021-08-30T04:07:46Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.us-east-1.amazonaws.com/": dial tcp 172.31.11.121:443: i/o timeout
2021-08-30T04:07:46Z W! 211 retries, going to sleep 1m0s before retrying.

Agent Log Error: Insufficient Permissions

2021-08-30T02:15:45Z E! cloudwatch: code: AccessDenied, message: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData, original error: 
2021-08-30T02:15:45Z W! 1 retries, going to sleep 400ms before retrying.
2021-08-30T02:15:46Z E! WriteToCloudWatch failure, err:  AccessDenied: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: cloudwatch:PutMetricData
    status code: 403, request id: f1171fd0-05b6-4f7d-bac2-629c8594c46e

Confirm connectivity to the CloudWatch endpoints

If traffic to CloudWatch doesn't go through the public internet, then you can use Amazon VPC endpoints. If you use Amazon VPC endpoints, then check the following parameters:

• If you use private name servers, then confirm that the DNS resolution provided accurate responses.
• Confirm that the CloudWatch endpoints resolve to private IP addresses.
• Confirm the security group that is associated with the Amazon VPC endpoint allows inbound traffic from the host.

To confirm connectivity to the CloudWatch endpoints, complete the following steps:

1. To check connectivity to the metrics endpoint, run the following command:

   $ telnet monitoring.us-east-1.amazonaws.com 443
   Trying 52.46.138.115...
   Connected to monitoring.amazonaws.com.
   Escape character is '^]'.
   ^]
   telnet> quit
   Connection closed.

2. To check connectivity to the logs endpoint, run the following command:

   $ telnet logs.us-east-1.amazonaws.com 443
   Trying 3.236.94.218...
   Connected to logs.us-east-1.amazonaws.com.
   Escape character is '^]'.
   ^]
   telnet> quit
   Connection closed

3. To check that the Amazon VPC endpoint resolves to private IP addresses, run the following command:

   $ dig monitoring.us-east-1.amazonaws.com 
   +short172.31.11.121
   172.31.0.13

Review the unified CloudWatch agent configuration

The CloudWatch agent configuration file details the metrics and logs that are published to CloudWatch. Review the agent configuration file to confirm that the logs and metrics that you want to publish are included.

Confirm that the host has permissions to publish metrics and logs

The AWS managed policies CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy can help you deploy the unified CloudWatch agent. The policies can also help check that you have the correct permissions. Use these policies as a reference to make sure that your host has the correct permissions.

The AWS CLI outputs in these examples show insufficient permissions.

The following AWS CLI config command shows that the AWS Identity and Access Management (IAM) role that is attached to the EC2 instance is missing:

$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Fail to fetch/remove json config: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Fail to fetch the config!

The following AWS CLI config command shows that the incorrect IAM role is attached to the EC2 instance:

$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:CWT-Web-Server -s
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:CWT-Web-Server --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Region: us-east-1
credsConfig: map[]
Error in retrieving parameter store content: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d
Fail to fetch/remove json config: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/cwagent/i-0744de7c842d2c2ba is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:123456789012:parameter/CWT-Web-Server
    status code: 400, request id: b85b0a7a-0fb1-47b4-924f-be8cf43a3b4d

Fail to fetch the config!

The following get-caller-identity command returns the IAM user or role that's associated with the instance:

$ aws sts get-caller-identity
{
    "UserId": "AROA123456789012ABCDE:i-0744de7c842d2c2ba",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/CloudWatchAgentServerRole/i-0744de7c842d2c2ba"
}

Confirm that the agent correctly starts

You can use AWS CLI with the configuration file passed as an argument to start the agent. To start the agent, run the following valid start commands.

For Linux, run the following commands:

- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path`
- `$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`

For Windows, run the following commands:

- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json"`
- `& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c ssm:configuration-parameter-store-name`

Important: Don't start the agent from the Windows Control Panel.

Confirm that the agent runs

To publish metrics and logs, the agent must be active. To verify that the CloudWatch agent is active, run the following command:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
    "status": "running",
    "starttime": "2021-08-30T02:13:44+00:00",
    "configstatus": "configured",
    "cwoc_status": "stopped",
    "cwoc_starttime": "",
    "cwoc_configstatus": "not configured",
    "version": "1.247349.0b251399"
}

Restart the agent after you update the agent configuration

The agent doesn't automatically register changes to the configuration file. If the agent configuration is updated to include new or different metrics and logs, then you must use the following command to restart the agent:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
****** processing cwagent-otel-collector ******
cwagent-otel-collector has already been stopped

****** processing amazon-cloudwatch-agent ******
Redirecting to /bin/systemctl stop amazon-cloudwatch-agent.service


$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/08/31 02:45:37 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting run_as_user...
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
{
  "status": "running",
  "starttime": "2021-08-31T02:45:37+0000",
  "configstatus": "configured",
  "cwoc_status": "stopped",
  "cwoc_starttime": "",
  "cwoc_configstatus": "not configured",
  "version": "1.247349.0b251399"
}

Related information

How do I install and configure the unified CloudWatch agent to send metrics and logs from my EC2 instance to CloudWatch?