Why isn't Systems Manager showing my Amazon EC2 instance as a managed instance?

9 minute read

I have an Amazon Elastic Compute Cloud (Amazon EC2) instance, but it doesn't appear as a managed instance in AWS Systems Manager.

Short description

To determine why AWS Systems Manager doesn't show your instance as managed, you can use the AWSSupport-TroubleshootManagedInstance runbook. For more information, see Run an automation operation powered by Systems Manager Automation and Setting up Automation.

You can also manually troubleshoot your Amazon EC2 instance.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Run the Systems Manager automation runbook

Prerequisites: Install and run AWS Systems Manager Agent (SSM Agent) on your instance. Before you run the automation, make sure that your AWS Identity and Access Management (IAM) user or role has the necessary permissions. See the Required IAM permissions section on AWSSupport-TroubleshootManagedInstance.

To run the runbook, complete the following steps:

Open the Systems Manager console.
In the navigation pane, choose Documents.
In the search bar, enter AWSSupport-TroubleshootManagedInstance (Owner: Amazon).
Choose the AWSSupport-TroubleshootManagedInstance document.
Choose Execute automation.
For the input parameters, enter the following information:
For InstanceId, enter the ID of the affected instance. You can manually enter the instance ID, or you can use the interactive instance picker. If you use the instance picker, then change the filter from Show managed instance only to Show all instances.
(Optional) For AutomationAssumeRole, enter the ARN of the IAM role that allows Systems Manager Automation to perform actions on your behalf. If you don't specify a role, then Systems Manager Automation uses the permissions of the user that runs the document.
Choose Execute.

After the automation completes, review the Outputs section for the following detailed results:

The InstanceisOnline.output section shows whether Systems Manager is managing the instance.
The FinalOutput.output section shows whether a check passed or failed and includes information about how to troubleshoot a failure.

Manually troubleshoot your Amazon EC2 instance

Important: Throughout the troubleshooting steps, select the AWS Region where your instance is located.

Verify that SSM Agent is installed and running on the instance

After you confirm that your operating system (OS) supports Systems Manager, verify that SSM Agent is installed and running on your instance.

SSM Agent is preinstalled on some Linux, macOS, and Windows Amazon Machine Images (AMIs).

To manually install SSM Agent when the agent isn't preinstalled, see the following AWS documentation:

To verify that SSM Agent is running, run the command that's specific to your OS to check the agent status.

After you verify that SSM Agent is running, run the ssm-cli command to troubleshoot managed instance availability.

Verify connectivity to Systems Manager endpoints on port 443

Connectivity verification to Systems Manager endpoints on port 443 is specific to your OS and subnet settings. For a list of Systems Manager endpoints by Region, see Service endpoints.

Note: In the following examples, the ssmmessages endpoint is required for Session Manager, a capability of AWS Systems Manager.

EC2 Linux instances

Use either Telnet or Netcat commands to verify connectivity to endpoints on port 443 for EC2 Linux instances. Netcat isn't preinstalled on EC2 instances. To manually install Netcat, see Ncat on the Nmap website.

Note: In the following commands, replace RegionID with your instance's Region ID.

Telnet commands:

telnet ssm.RegionID.amazonaws.com 443
telnet ec2messages.RegionID.amazonaws.com 443
telnet ssmmessages.RegionID.amazonaws.com 443

Example Telnet connection:

root@111800186:~# telnet ssm.us-east-1.amazonaws.com 443
Trying 52.46.141.158...
Connected to ssm.us-east-1.amazonaws.com.
Escape character is '^]'.

To exit from telnet, press the Ctrl and ] keys. Enter quit, and then press Enter.

Netcat commands:

nc -vz ssm.RegionID.amazonaws.com 443
nc -vz ec2messages.RegionID.amazonaws.com 443
nc -vz ssmmessages.RegionID.amazonaws.com 443

EC2 Windows instances

To verify connectivity to endpoints on port 443 for EC2 Windows instances, run the following Windows PowerShell commands:

`Test-NetConnection ssm.RegionID.amazonaws.com -port 443 Test-N`etConnection ec2messages.RegionID.amazonaws.com -port 443
Test-NetConnection ssmmessages.RegionID.amazonaws.com -port 443

Public subnets

Systems Manager endpoints are public. To resolve connectivity issues with instances in a public subnet, your instance's route table must route internet traffic to an internet gateway. Also, your Amazon Virtual Private Cloud (Amazon VPC) security groups and network access control lists (network ACLs) must allow outbound connections on port 443.

Private subnets

Use private IP addresses to privately access Amazon EC2 and Systems Manager APIs. To resolve connectivity issues with an instance in a private subnet, your instance's route table must route internet traffic to a NAT gateway. Or, you must configure your VPC endpoint to reach Systems Manager endpoints.

For more information, see How do I create VPC endpoints so that I can use Systems Manager to manage private EC2 instances without internet access?

Note: Each interface endpoint creates an elastic network interface in the provided subnet.

As a security best practice for private subnets, verify the following settings:

The security group that's attached to your VPC endpoint's network interface allows TCP port 443 inbound traffic from the security group that's attached to your instance.
The security group that's attached to your instance allows TCP port 443 outbound traffic to the private IP address for your VPC endpoint's network interface.

Verify the setup for Default Host Management Configuration

Note: If you didn't activate Default Host Management Configuration, then proceed to the Verify that the correct IAM role is attached to the instance section.

If the IAM role that Default Host Management Configuration created doesn't have sufficient permissions for your use case, then you can add policies.

All the associated instances must use Instance Metadata Service Version 2 (IMDSv2). To check your IMDSv2 configuration, use the MetadataNoToken Amazon CloudWatch metric to determine when there's zero IMDSv1 usage. Then, check if your instances are transitioned to IMDSv2.

Default Host Management Configuration is available in SSM Agent version 3.2.582.0 or later. To verify your SSM Agent version, see Checking the SSM Agent version number.

To verify the setup for Default Host Management Configuration, use either the Systems Manager console or the AWS CLI.

Systems Manager console

Complete the following steps:

Open the Systems Manager console.
In the navigation pane, choose Fleet Manager.
On the Account management dropdown menu, choose Default Host Management Configuration.
Verify that the Enable Default Host Management Configuration setting is turned on.

AWS CLI

Run the get-service-setting AWS CLI command to verify the setup for Default Host Management Configuration:

aws ssm get-service-setting \
--setting-id arn:aws:ssm:RegionID:AccountID:servicesetting/ssm/managed-instance/default-ec2-instance-management-role

Note: Replace AccountID with your AWS account ID.

After you activate Default Host Management Configuration, you receive an output that's similar to the following:

{
  "ServiceSetting": {
    "SettingId": "/ssm/managed-instance/default-ec2-instance-management-role",
    "SettingValue": "service-role/AWSSystemsManagerDefaultEC2InstanceManagementRole",
     "LastModifiedDate": 1679492424.738,
    "LastModifiedUser": "arn:aws:sts::012345678910:assumed-role/role/role-name",
    "ARN": "arn:aws:ssm:ap-southeast-1:012345678910:servicesetting/ssm/managed-instance/default-ec2-instance-management-role",
    "Status": "Customized"
  }
}

Note: If the value for SettingValue is $None, then Default Host Management Configuration isn't configured.

Verify that Default Host Management Configuration is using an appropriate IAM role

It's a best practice to use AWSSystemsManagerDefaultEC2InstanceManagementRole IAM when you set up Default Host Management Configuration. To use a different role, make sure to attach the AmazonSSMManagedEC2InstanceDefaultPolicy IAM policy to the role.

If you attached instance profiles to your instances, then remove any permissions that allow the ssm:UpdateInstanceInformation operation. SSM Agent tries to use instance profile permissions before it uses the Default Host Management Configuration permissions. When you allow the ssm:UpdateInstanceInformation operation in your instance profiles, your instance doesn't use the Default Host Management Configuration permissions.

Verify that the correct IAM role is attached to the instance

Note: If you activated Default Host Management Configuration, then proceed to the Verify connectivity to IMDS section.

To make API calls to a Systems Manager endpoint, you must attach the AmazonSSMManagedInstanceCore policy to the IAM role that's attached to your instance. If you're using a custom IAM policy, then confirm that your custom policy uses the permissions in AmazonSSMManagedInstanceCore. Also, make sure that the trust policy for your IAM role allows ec2.amazonaws.com to assume the role. For more information, see Alternative configuration for EC2 instance permissions.

Verify connectivity to IMDS

SSM Agent must communicate with IMDS to get information about your instance. To test the connection, run the following Netcat command:

nc -vz 169.254.169.254 80

To verify that IMDS is set up for your existing instance, use either the Amazon EC2 console or the AWS CLI.

Amazon EC2 console

Complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, choose Instances, and then select your instance.
Choose Actions, and then choose Instance settings.
Choose Modify instance metadata options.
In the dialog box, make sure that Instance metadata service is Enabled.

AWS CLI

Run the describe-instances AWS CLI command to verify that IMDS is set up for your existing instance:

aws ec2 describe-instances --query "Reservations[*].Instances[*].MetadataOptions" --instance-ids i-012345678910

Example output:

[
  [
    {
      "State": "applied",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    }
  ]
]

Note: If the output shows "HttpTokens": "optional", then both IMDSv1 and IMDSv2 are supported. If the output shows "HttpTokens": "required", then only IMDSv2 is supported. If the output shows "HttpEndpoint": "enabled", then IMDS is turned on.

If you're using a proxy on your instance, then the proxy might block connectivity to the metadata URL. To prevent the block, configure your SSM Agent to work with a proxy and set no_proxy for the metadata URL.

To configure SSM Agent to use a proxy, see the following AWS documentation:

Additional troubleshooting

If your instance still doesn't appear as a managed node or shows a lost connection in Systems Manager, then review the SSM Agent logs to continue to troubleshoot. For Linux and macOS, the logs are in /var/log/amazon/ssm. For Windows, the logs are in %PROGRAMDATA%\Amazon\SSM\Logs.

If your instance isn't reporting to SSM Agent, then use Remote Desktop Protocol (RDP) for Windows or SSH for Linux to collect the logs. If you can't collect the logs, then stop your instance and detach the root volume. Then, attach the root volume to another instance in the same Availability Zone as a secondary volume to get the logs.