How do I troubleshoot Session Manager errors?

4 minute read

I want to troubleshoot errors that I encounter when I use Session Manager, a capability of AWS Systems Manager.

Resolution

If a session fails, then follow the instructions at Troubleshooting Session Manager for comprehensive diagnostic guidance. If your Amazon Elastic Compute Cloud (Amazon EC2) instance isn't available as a managed Amazon EC2 instance, then troubleshoot your managed instance availability.

If a session fails and shows one of the following error messages, then apply the appropriate troubleshooting guidance.

Troubleshoot "Unable to retrieve data key" errors

The following error shows that users and EC2 instances in your account don't have the required AWS Key Management Service (AWS KMS) key permissions:

"Your session has been terminated for the following reasons: ----------ERROR------- Encountered error while initiating handshake. Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to an AWS KMS key that doesn't exist, doesn't exist in this Region, or that your IAM permissions don't allow you to access. status code: 400"

To resolve this error, activate AWS KMS encryption for your session data and grant the required AWS KMS key permissions to both users and instances.

Note: In AWS Systems Manager Agent (SSM Agent) version 3.2.582.0 and newer, Default Host Management Configuration automatically manages EC2 instances without an AWS Identity and Access Management (IAM) instance profile. The instances must use Instance Metadata Service Version 2 (IMDSv2).

Troubleshoot "WebSocket connection closed unexpectedly" errors

The following error indicates that the instance profile role attached to the target instance is missing kms:Decrypt permissions:

"Error - Fleet Manager is unable to start the session because the WebSocket connection closed unexpectedly during the handshake. Verify that your instance profile has sufficient Sessions Manager and AWS KMS permissions. For a more detailed message, visit the Session Manager console"

Systems Manager requires kms:Decrypt permissions for customer-managed key encryption and decryption of session data.

To resolve this error, add the following permissions to the instance profile role attached to your instance:

{ 
    "Effect":   "Allow", 
    "Action":  [ 
          "kms:Decrypt" 
    ], 
    "Resource": "arn:aws:kms:example-region:example-id:key/example-key-name" 
}

Note: Replace example-region with your AWS Region, example-id with your AWS account ID and example-key-name with your key. For more Session Manager permissions example, see Add Session Manager permissions to an existing IAM role.

Troubleshoot "Unable to validate encryption on Amazon S3 bucket" errors

The following error indicates that you chose Allow only encrypted S3 buckets for S3 logging in your Session Manager preferences:

"Your session has been terminated for the following reasons: Couldn't start the session because we are unable to validate encryption on Amazon S3 bucket. Error: AccessDenied: Access Denied status code: 403"

To resolve this error, turn off the Allow only encrypted S3 buckets option in your Session Manager preferences and configure the required IAM permissions. For instructions, see Enabling and disabling session logging.

Troubleshoot "Encryption is not set up on the selected CloudWatch Logs log group" errors

The following error indicates that you chose Allow only encrypted CloudWatch log groups for CloudWatch logging in your Session Manager preferences:

"Your session has been terminated for the following reasons: We couldn't start the session because you haven't set up encryption on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to turn on logging without encryption"

To resolve this error, turn off the Allow only encrypted CloudWatch log groups option in your Session Manager preferences and configure the required IAM permissions.

Troubleshoot "Failed to create user ssm-user" errors

The following error shows that you use Systems Manager for Windows Server on a domain controller without a ssm-user account:

"Your session has been terminated for the following reasons: ----------ERROR------- Unable to start command: Failed to create user ssm-user: Instance is running active directory domain controller service. Disable the service to continue to use session manager"

If you use SSM agent version 2.3.612.0 and newer, then you must manually create the ssm-user account that Windows Server machines use as domain controllers. Be sure to grant the ssm-user user account the required permissions.

To identify the SSM Agent version, see Checking the SSM Agent version number. Also review the SSM Agent release notes at RELEASENOTES.md on the GitHub website.

To subscribe to SSM Agent notification, see Subscribing to SSM Agent notifications.

To configure SSM Agent to automatically update, see Automating updates to SSM Agent.