How do I configure instances in an EC2 Auto Scaling group that doesn't have internet access to send metrics and logs to CloudWatch?

Resolution

To configure EC2 instances in an EC2 Auto Scaling group to send metrics and logs to CloudWatch, take the following actions.

Install the CloudWatch agent

Install the CloudWatch agent on an instance that has internet connectivity. Or, use an existing instance that already sends logs and metrics to CloudWatch with the CloudWatch agent.

Note: Include the Auto Scaling group name as a dimension in your CloudWatch agent configuration. The agent retrieves this name from instance tags and other metadata, such as ImageId, InstanceId, and InstanceType.

Then, verify that the CloudWatch agent successfully sends metrics and logs to CloudWatch.

Create a launch template for your EC2 Auto Scaling group

Create a launch template for the EC2 Auto Scaling group.

Choose Advanced details. Then, choose IAM instance profile and User data. For User data, enter a script to install and configure the CloudWatch agent based on the JSON that you configured. For IAM instance profile, select the AWS Identity and Access Management (IAM) role that allows EC2 instances to send metrics and logs to CloudWatch. For information about these advanced settings, see Advanced settings.

Example user data script on an instance:

#!/bin/bashcd /tmp
wget https://s3.REGION.amazonaws.com/amazoncloudwatch-agent-REGION/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm
cat << EOF > /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "append_dimensions": {
      "InstanceId": "\${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "disk_used_percent"
        ],
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      }
    }
  }
}
EOF
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Note: Replace REGION with your AWS Region. To escape special characters inside the config.json file, use backslash (\). The script configures the CloudWatch agent to monitor memory and disk utilization, and then starts the agent. Make sure to use the download link for your EC2 Auto Scaling group's Region.

Create VPC endpoints for your metrics and logs

Create interface virtual private cloud (VPC) endpoints for CloudWatch metrics and Amazon CloudWatch Logs to the VPC that hosts the private subnets. Then, update your endpoint policies with custom policies to allow the required CloudWatch actions. For information about service endpoints, see Amazon CloudWatch endpoints and quotas.

Note: Make sure to use the endpoint that corresponds with your EC2 Auto Scaling group's Region. For example, if your EC2 Auto Scaling group is in the eu-west-2 Region, then use monitoring.eu-west-2.amazonaws.com for metrics and logs.eu-west-2.amazonaws.com for logs.

Example CloudWatch VPC endpoint policy:

{  "Statement": [
    {
      "Sid": "PutOnly",
      "Principal": "*",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Example CloudWatch Logs VPC endpoint policy:

{  "Statement": [
    {
      "Sid": "PutOnly",
      "Principal": "*",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Then, create a VPC gateway endpoint for Amazon Simple Storage Service (Amazon S3) to the VPC that hosts your private subnets. This endpoint allows instances in the private subnets to download the CloudWatch agent package.

Create an EC2 Auto Scaling group

Create an EC2 Auto Scaling group with private subnets activated from the launch template that you created.