Unable to use Session Manager on EC2 instances in a private subnet with SSM VPC endpoint

Question

I am setting up an environment to mimic what customer wants to achieve. I have EC2 instances in a private subnet in a VPC. In order to use Session Manager on them, I created VPC endpoint to allow SSM communication. Those EC2 instances has instance profile with an IAM role granting managed policy " AmazonSSMManagedInstanceCore".

All the instances are showing up properly in Systems Manager. However, when I tried to start a session using Session Manager, when I select the instance, it shows the following error message:

> The version of SSM Agent on the instance supports Session Manager, but
> the instance is not configured for use with AWS Systems Manager.
> Verify that the IAM instance profile attached to the instance includes
> the required permissions.

To compare and troubleshoot, I launched EC2 instances in a public subnet, using the same IAM role, they all working well with session manager. The ssm-agent version on those EC2 instances are 2.3.662.0 and 2.3.372.0, all supported for Session Manager. The only difference between working and non-working instances are the working ones are running from public subnet, while the non-working ones are running from private subnet with SSM VPC endpoint.

What could be wrong? Thanks

Accepted Answer

Make sure that you have specified all VPC endpoint for SSM:

- **com.amazonaws.region.ssm**: The endpoint for the Systems Manager
   service.
 - **com.amazonaws.region.ec2messages**: Systems Manager uses this endpoint
   to make calls from SSM Agent to the Systems Manager service.
 - **com.amazonaws.region.ec2**: If you're using Systems Manager to create
   VSS-enabled snapshots, you need to ensure that you have an endpoint
   to the EC2 service. Without the EC2 endpoint defined, a call to
   enumerate attached EBS volumes fails, which causes the Systems
   Manager command to fail.
 **- com.amazonaws.region.ssmmessages**: This endpoint is required only if
   you are connecting to your instances through a secure data channel
   using Session Manager. For more information, see AWS Systems Manager
   Session Manager.

Source: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html#sysman-setting-up-vpc-create

Answer

I followed all docos available under the sun: all possible SG to protect instance and/or VPC endpoint.
It only worked once (Connect button was available, and I could open a session onto instance).
Then I followed the advice to restrict the Source CIDR of VPC endpoint Inbound SG to priv subnet, (instead of entire VPC), and it failed with error: "SSM Agent is offline".
When I rolled back SG to entire VPC, it never worked again...

The only way I could make it work is by adding a **NAT Gwy**.
I anyway like NAT Gwy to keep my EC2 up to date in terms of patching level.

Conclusion : Total fiasco, and 6 hours wasted. NAT Gwy fixed it and allows decent security level of instance.

Unable to use Session Manager on EC2 instances in a private subnet with SSM VPC endpoint

Conteúdo relevante