Unable to use Session Manager on EC2 instances in a private subnet with SSM VPC endpoint

0

I am setting up an environment to mimic what customer wants to achieve. I have EC2 instances in a private subnet in a VPC. In order to use Session Manager on them, I created VPC endpoint to allow SSM communication. Those EC2 instances has instance profile with an IAM role granting managed policy " AmazonSSMManagedInstanceCore".

All the instances are showing up properly in Systems Manager. However, when I tried to start a session using Session Manager, when I select the instance, it shows the following error message:

The version of SSM Agent on the instance supports Session Manager, but the instance is not configured for use with AWS Systems Manager. Verify that the IAM instance profile attached to the instance includes the required permissions.

To compare and troubleshoot, I launched EC2 instances in a public subnet, using the same IAM role, they all working well with session manager. The ssm-agent version on those EC2 instances are 2.3.662.0 and 2.3.372.0, all supported for Session Manager. The only difference between working and non-working instances are the working ones are running from public subnet, while the non-working ones are running from private subnet with SSM VPC endpoint.

What could be wrong? Thanks

asked 5 years ago6588 views
2 Answers
1
Accepted Answer

Make sure that you have specified all VPC endpoint for SSM:

  • com.amazonaws.region.ssm: The endpoint for the Systems Manager service.
  • com.amazonaws.region.ec2messages: Systems Manager uses this endpoint to make calls from SSM Agent to the Systems Manager service.
  • com.amazonaws.region.ec2: If you're using Systems Manager to create VSS-enabled snapshots, you need to ensure that you have an endpoint to the EC2 service. Without the EC2 endpoint defined, a call to enumerate attached EBS volumes fails, which causes the Systems Manager command to fail. - com.amazonaws.region.ssmmessages: This endpoint is required only if you are connecting to your instances through a secure data channel using Session Manager. For more information, see AWS Systems Manager Session Manager.

Source: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html#sysman-setting-up-vpc-create

answered 5 years ago
profile picture
EXPERT
reviewed 24 days ago
profile picture
EXPERT
reviewed 2 months ago
  • Also, I'm still confused if a VPC endpoint is just like a wormhole between the VPC and AWS Services, which will avoid packets to and from the instance to travel over the Internet?

  • The documentation referenced is not clear enough. I still don't know which type of endpoint I need, in the 1st page of the creation wizard, among: AWS Services, EC2 Instance Connect Endpint, PrivateLink, and possibly others. Also, you'll note the black magic that consists in inverting the Service Name into a namespace to be "verified" with some types, not others. The comment above uses the namespace notation, which, in particular, is valid for PrivateLink type, but not only.

0

I followed all docos available under the sun: all possible SG to protect instance and/or VPC endpoint. It only worked once (Connect button was available, and I could open a session onto instance). Then I followed the advice to restrict the Source CIDR of VPC endpoint Inbound SG to priv subnet, (instead of entire VPC), and it failed with error: "SSM Agent is offline". When I rolled back SG to entire VPC, it never worked again...

The only way I could make it work is by adding a NAT Gwy. I anyway like NAT Gwy to keep my EC2 up to date in terms of patching level.

Conclusion : Total fiasco, and 6 hours wasted. NAT Gwy fixed it and allows decent security level of instance.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions