I want to troubleshoot errors when using custom launch template with managed node groups in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.
Short description
When you use a custom launch template with managed node groups in your Amazon EKS cluster, you might get one of the following errors:
- "Client.InternalError: Client error on launch"
- "SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-xxxxxxxxxxxxxxxxx version x was not found"
- "Node group entered a DEGRADED status"
Resolution
Client.InternalError: Client error on launch
This error occurs when you use encrypted Amazon Elastic Block Store (Amazon EBS) volumes in the custom launch template with incorrect permissions. To check the encryption status and AWS Identity and Access Management (IAM) permissions or policies, complete the following tasks:
SourceEc2LaunchTemplateNotFound or The Amazon EC2 Launch Template lt-xxxxxxxxxxxxxxxxx version x was not found
This error occurs when you manually change the custom launch template version through the Auto Scaling group of the node instead of Amazon EKS. To resolve this issue, you must use Amazon EKS to update the launch template version.
To update your EC2 launch template from the managed node group, complete the following steps:
- Open the Amazon EKS console.
- In the navigation pane, choose Clusters.
- Under Cluster name, choose the cluster that contains the node group to update.
- Choose the Compute tab.
- For Node groups, choose your node and then choose Change version.
- Select the version to apply to your node group. Make sure that the update strategy is set to Rolling Update.
- Choose Update.
This error also occurs if you delete a reference template when you create a node group with a custom launch template. When you create a node group with a custom launch template, Amazon EKS replicates the template. If you delete the original template, then you must recreate the node group.
To recreate the node group, complete the following steps:
-
Launch a new node group.
-
Run the following command to verify that all nodes are healthy, in the Ready state, and joined the cluster:
$ kubectl get nodes
-
Drain your worker nodes.
-
Delete the original node group.
The Node group enters a DEGRADED status after it creates a new launch template
A node group can enter a DEGRADED status with a message similar to the following error:
"The Amazon EC2 Launch Template : lt-xxxxxxxxxxxxxxxxx has a new version associated with your Autoscaling group, which is not managed by Amazon EKS. Expected Launch Template version: x".
This error occurs when the Amazon EC2 launch template version for your managed node group doesn't match the version that Amazon EKS creates. Existing node groups that don't use a custom launch template can't be directly updated. To resolve this error, create a launch template and version with your preferred settings. Then, use the launch template to create the node group. If the new node group is launched from your custom template, then create new versions of the template. You can use this template without placing the node group in a DEGRADED status.
Related information
Customizing managed nodes with launch templates
Launch template configuration basics