When I start my Amazon EC2 instance that has encrypted volumes attached, why do I get the "client error on launch" error?

Short description

After you launch an Amazon EC2 instance that has encrypted volumes attached, you might have problems with the instance starting. The instance goes from a pending state to shutting down, and then to a terminated state. If you run the AWS Command Line Interface (AWS CLI) describe-instances command, you might see an error similar to the following:

" "StateReason": { "Code": "Client.InternalError" "Message": "Client.InternalError: Client error on launch" }"

This problem occurs with EC2 instances with encrypted volumes in the following situations:

• The AWS Key Management Service (AWS KMS) or AWS Identity and Access Management (IAM) principal launching the instances doesn't have the required permissions.
• The KMS key usage is restricted by an incompatible condition, such as the SourceIp condition key.

The IAM principal used to start the instance must have permission to invoke the CreateGrant action on the KMS key.

To allow access to a KMS key, you must use the key policy in combination with IAM policies or grants.

By default, AWS KMS key policies grant access to the account principal and allow you to use IAM policies to grant access to your keys. Even when EC2 full privilege is provided to an IAM user or role, AWS KMS permissions must grant access to the keys through one of the following methods:

• Explicitly name the principal (IAM user/role) in your key policy statements.
• Grant KMS permissions in an IAM policy while naming the account principal in your key policy.

Resolution

Create an IAM policy to allow the IAM principal to call AWS KMS APIs

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Complete the following steps:

1. Open the IAM console, choose Policies, and then choose Create policy.
2. Choose the JSON tab, and then enter the following policy into the JSON field:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": [
           "kms:Decrypt",
           "kms:ReEncrypt*",
           "kms:Encrypt",
           "kms:GenerateDataKey*",
           "kms:DescribeKey"
         ],
         "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
       },
       {
         "Effect": "Allow",
         "Action": "kms:CreateGrant",
         "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
         "Condition": {
           "Bool": {
             "kms:GrantIsForAWSResource": true
           }
         }
       }
     ]
   }

   Note: For Resource, use your key ARN.
3. Choose Review policy.
4. For Name, enter a name for the policy. Then, choose Create policy.
5. Choose the policy that you created.
6. Choose the Policy usage tab, and then choose Attach.
7. For Name, choose the IAM entity that you want to grant permissions to KMS key, and then choose Attach policy.

Grant the IAM principal explicit access to a KMS key

Complete the following steps:

1. Open the AWS KMS console, and choose Customer managed keys.
2. For Key ID, choose your Key ID.
3. For Key users, choose Add.
4. For Name, choose the IAM user or role, and then choose Add.
   Note: The default key policy allows IAM access for your accounts. If you use a custom key policy instead, then the KMS key must explicitly grant the following permissions:

   {
         "Sid": "Allow use of the key",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::123456789012:role/MyRoleName",
             "arn:aws:iam::123456789012:user/MyUserName"
           ]
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt*",
           "kms:GenerateDataKey*",
           "kms:DescribeKey"
         ],
         "Resource": "*"
       },
       {
         "Sid": "Allow attachment of persistent resources",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::123456789012:role/MyRoleName",
             "arn:aws:iam::123456789012:user/MyUserName"
           ]
         },
         "Action": [
           "kms:CreateGrant",
           "kms:ListGrants",
           "kms:RevokeGrant"
         ],
         "Resource": "*",
         "Condition": {
           "Bool": {
             "kms:GrantIsForAWSResource": "true"
           }
         }
       }

Use the IP address condition

If you use AWS KMS to protect your data for an integrated service, then review your access policy statement. There can be problems when you specify the IP address condition operators or the aws:SourceIp condition key in the same access policy statement. When you attach an encrypted Amazon Elastic Block Store (Amazon EBS) volume to an EC2 instance, Amazon EC2 sends a request to AWS KMS. The request decrypts the volume's encrypted data key. This request comes from an IP address associated with the EC2 instance and not the user's IP address. This means that the decryption request is rejected if you have a SourceIp condition set. Then, the instance fails.

Instead, use the kms:ViaService condition key. AWS KMS allows interactions from that service on your behalf. The principal role has permission to use the KMS key and integrated service. For more information, see kms:ViaService condition key limits.

Note: EC2 instances that have users that are logged in can't interact with this condition. Only the service that can act on your behalf can interact with the condition. This interaction is logged in AWS CloudTrail logs for you to review.

In the following example, the CloudTrail entry for an API call is made to AWS KMS. This is called on by Amazon EC2, not from a specific IP address. When you add a policy to a user that allows AWS KMS to interact with Amazon EC2, then the API call can complete.

Example:

"userIdentity": {
  "sessionContext": {
  "sessionIssuer": {
    "accountId": "123456789012",
    "principalId": "123456789012:aws:ec2-infrastructure",
    "userName": "aws:ec2-infrastructure",
    "arn": "arn:aws:iam::123456789012:role/aws:ec2-infrastructure",
    "type": "Role"
   },
...
  "eventType": "AwsApiCall",
  "@log_group": "CloudTrail/AllRegionLogGroup",
  "awsRegion": "us-west-2",
  "requestParameters": {
    "encryptionContext": {
    "aws:ebs:id": "vol-0ca158925aa9c1883"
    }
}

Related information

Condition keys for AWS KMS

How can I verify that authenticated encryption with data encryption is used for AWS KMS API calls?