How do I troubleshoot custom launch template issues with managed node groups in Amazon EKS?

Short description

When you use a custom launch template with managed node groups in your Amazon EKS cluster, you might get one of the following errors:

• "Client.InternalError: Client error on launch"
• "SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-xxxxxxxxxxxxxxxxx version x was not found"
• "Node group entered a DEGRADED status"

Resolution

Client error on launch

This error occurs when you use encrypted Amazon Elastic Block Store (Amazon EBS) volumes in the custom launch template with incorrect permissions. To check the encryption status and AWS Identity and Access Management (IAM) permissions or policies, complete the following tasks:

• Activate the AWS Key Management Service (AWS KMS) key that's used to encrypt the volume.
• Make sure that the AWS KMS key has the correct key policies. To allow more IAM roles to work with encrypted Amazon EBS volumes, modify the key policies. For more information, see Allows key users to use the KMS key.

Launch template not found

This error occurs when you manually change the custom launch template version through the Auto Scaling group of the node instead of Amazon EKS. To resolve this issue, you must use Amazon EKS to update the launch template version.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To update your Amazon Elastic Compute Cloud (Amazon EC2) launch template from the managed node group, complete the following steps:

1. Open the Amazon EKS console.
2. In the navigation pane, choose Clusters.
3. Under Cluster name, select the cluster that contains the node group to update.
4. Choose the Compute tab.
5. Under Node groups, select your node, and then choose Change version.
   Note: You see this option only if there's a new revision of the launch template available. If you can't see the option, then use the aws eks update-nodegroup-version AWS CLI command to modify the launch template version of the node group.
6. Select the version to apply to your node group. Make sure that the update strategy is set to Rolling Update.
7. Choose Update.

This error also occurs if you delete a reference template when you create a node group with a custom launch template. If you use a custom launch template, then Amazon EKS replicates the template. If you delete the original template, then you must recreate the node group.

To recreate the node group, complete the following steps:

1. Launch a new node group.

2. Run the following command to verify that all nodes are healthy, in the Ready state, and joined the cluster:

   $ kubectl get nodes

3. Drain your worker nodes.

4. Delete the original node group.

Node group enters a DEGRADED status

A node group can enter a DEGRADED status after Amazon EKS creates a launch template with an error message similar to the following:

"The Amazon EC2 Launch Template : lt-xxxxxxxxxxxxxxxxx has a new version associated with your Autoscaling group, which is not managed by Amazon EKS. Expected Launch Template version: x".

This error occurs when the Amazon EC2 launch template version for your managed node group doesn't match the version that Amazon EKS creates. You can only directly update existing node groups that use a custom launch template. To resolve this error, create a launch template and version with your preferred settings. Then, use the launch template to create the node group. If the new node group launches from your custom template, then create new versions of the template. You can use this template without placing the node group in a DEGRADED status.

Related information

Customize managed nodes with launch templates

Launch template configuration basics