How do I troubleshoot and resolve CloudWatch agent metrics that don't appear in the console?

Short description

The CloudWatch agent collects system-level metrics from your Amazon EC2 instance and publishes them to CloudWatch. CloudWatch displays these metrics in the console after the agent successfully publishes them. Metrics might not appear in the following situations:

• The metric definition doesn't exist in the agent configuration file.
• The AWS Identity and Access Management (IAM) role attached to the EC2 instance doesn't have permissions to publish metrics to CloudWatch.
• The agent can't reach CloudWatch endpoints because of network or proxy issues.
• The agent isn't running.

Resolution

Verify the CloudWatch agent status

Run one of the following commands for your operating system (OS) to confirm that the CloudWatch agent runs on your instance:

Linux:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

Windows:

& $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -m ec2 -a status

If the agent runs correctly, then the output shows "status": "running" and "configstatus": "configured."

Example output:

{"status": "running","starttime": "2025-01-15T10:30:00+0000","configstatus": "configured","version": "1.247349.0b251399"}

If the output shows "stopped" or "not configured", then proceed to the Restart the agent after configuration changes section in this article.

Review the agent configuration file

The CloudWatch agent configuration file specifies the metrics and logs that the agent publishes to CloudWatch.

Open the configuration file and confirm that the metrics_collected section includes the expected metrics.

By default, the configuration file is in the following directory:

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Default configuration file path for Windows: C:\ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json

Note: If you specified a custom path when you installed the agent, then check that path instead. For more information about the configuration file, see CloudWatch agent configuration file: Metrics section.

For example, the following configuration collects memory and disk metrics:

{
  "metrics": {
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent"]
      },
      "disk": {
        "measurement": ["disk_used_percent"],
        "resources": ["*"]
      }
    }
  }
}

If the metric definition doesn't exist in the configuration file, then add it to the configuration file. Then, restart the CloudWatch agent. It's a best practice to store your CloudWatch agent configuration in Parameter Store, a capability of AWS Systems Manager. New instances can then fetch the configuration automatically during startup.

Restart the agent after configuration changes

The agent doesn't automatically register changes to the configuration file. After you update the configuration, run the following commands to stop and restart the agent with the new configuration.

Run the following command to stop the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop

Run the following command to restart the agent with the updated configuration:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Note: If the configuration file is stored in Parameter Store, then replace the file path with ssm:PARAMETER-STORE-NAME.

Run the following command to confirm the agent status:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

Check IAM permissions

Run the following describe-instances command to check the IAM role attached to your instance:

aws ec2 describe-instances --instance-ids INSTANCE_ID --query "Reservations[*].Instances[*].IamInstanceProfile.Arn" --output text

Note: Replace INSTANCE_ID with your instance ID.

Then, run the following list-attached-role-policies command to verify the attached policies:

aws iam list-attached-role-policies --role-name ROLE_NAME

Note: Replace ROLE_NAME with the role name from the previous command output.

Confirm that the CloudWatchAgentServerPolicy managed policy appears in the output.

If the output doesn't include the CloudWatchAgentServerPolicy policy, then run the following attach-role-policy command to attach the managed policy:

aws iam attach-role-policy --role-name ROLE_NAME --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Verify network connectivity to CloudWatch endpoints

The CloudWatch agent must reach CloudWatch endpoints to publish metrics. If your instance uses a proxy or virtual private cloud (VPC) endpoint, then verify the endpoint configuration. Then, test connectivity to CloudWatch endpoints.

Check proxy settings

Open the common-config.toml configuration file for your OS to check the proxy configuration.

Linux: /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Windows: C:\ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Verify that the file contains the proxy server URLs and the no_proxy setting.

Example proxy configuration:

[proxy]http_proxy = "HTTP-PROXY-URL"https_proxy = "HTTPS-PROXY-URL"no_proxy = "169.254.169.254,monitoring.REGION.amazonaws.com"

Note: Replace HTTP-PROXY-URL and HTTPS-PROXY-URL with your proxy URLs. Replace REGION with your AWS Region.

If the agent configuration uses an endpoint_override, then verify that the VPC endpoint URL is correct in the configuration file.

Example:

{"metrics": {"endpoint_override": "vpce-VPC_ENDPOINT.monitoring.REGION.vpce.amazonaws.com",...}}

Note: Replace VPC_ENDPOINT with your VPC endpoint ID. Replace REGION with your Region.

Test connectivity to CloudWatch endpoints

Run one of the following commands for your OS to verify that the instance reaches CloudWatch endpoints.

Linux:

curl -v https://monitoring.REGION.amazonaws.com

Windows:

Test-NetConnection monitoring.REGION.amazonaws.com -Port 443 -InformationLevel Detailed

Note: In the preceding commands, replace REGION with your Region.

If the connection fails, then verify that security group rules and network access control lists (network ACLs) allow outbound traffic on port 443 to CloudWatch endpoints.

Review CloudWatch agent logs for errors

The CloudWatch agent log file is in the following directory:

• Linux: /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
• Windows: $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Open the log file and search for "error" or "AccessDenied" messages.

If the agent can't reach the CloudWatch endpoint, then you might receive the following error:

"net/http: request canceled while waiting for connection"

If the agent doesn't have the required IAM permissions, then you might receive the following error:

"AccessDenied: User: arn:aws:sts::123456789012:assumed-role/role-name is not authorized to perform: cloudwatch:PutMetricData"

Verify that the metric exists in CloudWatch

After you confirm the agent runs and the configuration is correct, verify that the metric appears in CloudWatch.

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose All metrics.
3. Search for the CWAgent namespace.

If the metric appears in the CWAgent namespace, then the CloudWatch agent successfully publishes metrics to CloudWatch.

If the metric doesn't appear, then run the following list-metrics command to check whether CloudWatch received the metric:

aws cloudwatch list-metrics --namespace CWAgent --metric-name METRIC_NAME

Note: Replace METRIC_NAME with the name of your metric.

If the command returns results, then the metric exists but might not appear in the console because the console filters by specific dimensions. To resolve this issue, either update alarm dimensions or aggregate metrics by InstanceId. For instructions, see the Resolve dimension mismatches section in How do I resolve the "Insufficient Data" error for CloudWatch alarms that monitor disk and memory usage for EC2 instances?

If the command returns no results, then the agent doesn't publish the metric. Review the agent configuration file to confirm the metric definition exists, and then check the agent logs for errors.

Related information

CloudWatch agent files and locations