How do I troubleshoot issues when I set up Cluster Autoscaler on an Amazon EKS cluster?

10 minute read

I want to troubleshoot issues when I launch Cluster Autoscaler on my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

The commands in the following resolution don't use the --region flag because they use the default AWS Region. To check your default Region, run the following AWS CLI configure command:

aws configure

To change the Region, use the --region flag.

Prerequisites: Install or update eksctl to the latest version. For instructions, see Installation on the eksctl website.

Take the following troubleshooting actions based on the issue that you encounter. To troubleshoot probe issues, see How do I troubleshoot liveness and readiness probe issues with my Amazon EKS clusters?

Cluster Autoscaler Pod is in a "CrashLoopBackOff" state

To troubleshoot the "CrashLoopBackOff" state, complete the following steps:

To check the Cluster Autoscaler Pod status, run the following command:

kubectl get pods -n kube-system | grep cluster-autoscaler

Example output:

NAME                            READY   STATUS             RESTARTS      AGE
cluster-autoscaler-abcd-abcd   0/1     CrashLoopBackOff   3 (20s ago)   99s

To get details about why the Pod is stuck in the "CrashLoopBackOff" state, run the following command:

kubectl describe pod cluster-autoscaler-abcd-abcde -n kube-system

Note: Replace cluster-autoscaler-abcd-abcde with the name of the Cluster Autoscaler Pod.
In the output, check the value for Reason. If you see an OOMKilled issue, then use the kubectl edit command to increase the limits and requests values for memory resources in the Cluster Autoscaler deployment. For more information about kubectl edit, see kubectl edit on the Kubernetes website. To view the default memory values for the Cluster Autoscaler, see cluster-autoscaler-autodiscover.yaml on the GitHub website.
Example output:

Name:               cluster-autoscaler-abcd-abcde
Namespace:          kube-system
State:              Waiting
Reason:             CrashLoopBackOff
Last State:         Terminated
Reason:             OOMKilled
Exit Code:          137
...

To check for issues in the Cluster Autoscaler Pod logs, run the following command:

kubectl logs -f -n kube-system -l app=cluster-autoscaler

If the logs show AWS Identity and Access Management (IAM) permissions issues, then proceed to Cluster Autoscaler Pod has IAM permissions issues. Example output:

Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: abc is not authorized to perform: autoscaling: DescribeTags because no identity-based policy allows the autoscaling:DescribeTags action status code: 403, request id: abcdexyz

If the logs show networking issues such as I/O timeout, then proceed to Cluster Autoscaler Pod has networking issues. Example output:

Failed to create AWS Manager: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials caused by: RequestError: send request failed caused by: Post https://sts.region.amazonaws.com/: dial tcp: i/o timeout

Cluster Autoscaler Pod has IAM permissions issues

Check whether you associated an OIDC provider with the cluster

Make sure that you created an OpenID Connect (OIDC) provider for your cluster.

Check whether you annotated the Cluster Autoscaler service account with the IAM role

To check your AWS service account annotations, run the following command:

kubectl get serviceaccount cluster-autoscaler -n kube-system -o yaml

Check annotations to make sure that the output lists your Cluster Autoscaler service account role. Example output:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/cluster_auto_scaler_iam_role
  name: cluster-autoscaler
  namespace: kube-system

To troubleshoot issues with your service account role, see How do I troubleshoot an OIDC provider and IRSA in Amazon EKS?

Check the IAM policy

Make sure that you attached the correct IAM policy to the Cluster Autoscaler service account role. For an example policy and list of required permissions, see IAM policy on the GitHub website.

Check whether you correctly configured the trust relationship

Make sure that you correctly configured the trust relationship. Example trust relationship:

{  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::example_awsaccountid:oidc-provider/oidc.eks.example_region.amazonaws.com/id/example_oidcid"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.example_region.amazonaws.com/id/example_oidcid:aud": "sts.amazonaws.com",
          "oidc.eks.example_region.amazonaws.com/id/example_oidcid:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
        }
      }
    }
  ]
}

Note: Replace example_awsaccountid with your AWS account ID, example_region with your Region, and example_oidcid with your OIDC ID.

Each time that you change the service account role or policy, run the following command to restart the Cluster Autoscaler Pod:

kubectl rollout restart deployment cluster-autoscaler -n kube-system

Cluster Autoscaler Pod has networking issues

Make sure that you configured the Amazon EKS cluster with the required networking setup. Verify that the worker node subnet has a route table that can route traffic to the following endpoints:

ec2.amazonaws.com for Amazon Elastic Compute Cloud (Amazon EC2)
autoscaling.region.amazonaws.com for Amazon EC2 Auto Scaling
sts.region.amazonaws.com for AWS Security Token Service (AWS STS)
Note: Replace region with your Region.

If the Amazon EKS cluster is private, then you must create an Amazon Virtual Private Cloud (Amazon VPC) endpoint for the preceding endpoints.

Note: Each VPC endpoint's security group must allow the Amazon EKS worker node security group. The security groups must also allow inbound traffic to the Amazon EKS VPC CIDR block on port 443.

Verify that the subnet network access control list (network ACL) and the worker node security groups don't block traffic that communicates to these endpoints.

Cluster Autoscaler doesn't scale in or scale out nodes

Check the Cluster Autoscaler Pod logs for the scheduling rules

To check the status of your Pods, run the following command:

kubectl logs -f -n kube-system -l app=cluster-autoscaler

To check whether the Pod that's in a Pending status contains scheduling rules, such as the affinity rule, run the following command:

kubectl describe pod example_podname -n example_namespace

Note: Replace example_podname with the Pod name, and example_namespace with your namespace.

In the output, check the events section for information about why the Pod is in a Pending status. Example output:

$ kubectl describe pod cluster-autoscaler-abcde-abcd -n kube-system 
Name: cluster-autoscaler-5b6b675456-npf8p
...
Status: Pending
...
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning FailedScheduling 12s (x4 over 2m) default-scheduler 0/4 nodes are available: 1 node(s) didn't match node selector, 3 node(s) didn't match pod affinity rules.

Note: Cluster Autoscaler respects nodeSelector and requiredDuringSchedulingIgnoredDuringExecution in nodeAffinity. Make sure to label your node groups with these values. To label your node groups, run the following command:

kubectl get nodes --show-labels

If Cluster Autoscaler can't schedule a pod with nodeSelector or requiredDuringSchedulingIgnoredDuringExecution, then the Cluster Autoscaler only uses node groups that meet expansion requirements. To schedule a Pod on a node, you must modify the scheduling rules defined on the Pods or nodes. For more information, see Affinity and anti-affinity on the Kubernetes website.

Check the Amazon EC2 Auto Scaling group tagging for the Cluster Autoscaler

For the Cluster Autoscaler to discover an Auto Scaling group, you must tag the node group's Amazon EC2 Auto Scaling group with the following tags.

For tag 1, configure the following values:

For key, enter k8s.io/cluster-autoscaler/example-cluster.
For value, enter owned.

Note: Replace example-cluster with your cluster name.

For tag 2, configure the following values:

For key, enter k8s.io/cluster-autoscaler/enabled.
For value, enter true.

Check the configuration of the deployment manifest

To open the deployment manifest, run the following command:

kubectl -n kube-system edit deployment.apps/cluster-autoscaler

In the output, for node-group-auto-discover, make sure that the manifest lists the correct cluster or node group name. Example output:

containers:- command   ./cluster-autoscaler
   --v=4
   --stderrthreshold=info
   --cloud-provider=aws
   --skip-nodes-with-local-storage=false
   --expander=least-waste
   --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/example-cluster
   --balance-similar-node-groups
   --skip-nodes-with-system-pods=false

Make sure that you don't exceed your maximum number of nodes

To check the current number of nodes, run the following describe-nodegroup command:

aws eks describe-nodegroup --cluster-name example-cluster --nodegroup-name example-nodegroup

Note: Replace example-cluster with your cluster name, and example-nodegroup with your node group name.

If you reached the minimum or maximum number of nodes, then run the following update-nodegroup-config command to modify the values:

aws eks update-nodegroup-config \
    --cluster-name cluster-name \
    --nodegroup-name nodegroup-name \
    --scaling-config minSize=minvalue,maxSize=maxvalue,desiredSize=desiredvalue \

Note: Replace cluster-name with your cluster name, and nodegroup-name with your node group name. Also, replace minvalue with the minimum number of nodes, maxvalue with the maximum number of nodes, and desiredvalue with the desired value of nodes.

Make sure that your nodes can join the cluster

Check whether the new instances that the Amazon EC2 Auto Scaling group launches can join the Amazon EKS cluster. If they can't, then see How do I get my worker nodes to join my Amazon EKS cluster?

Make sure that you use the correct node instance type

To check whether Cluster Autoscaler can fulfill the Pod resource request with the current node instance types, run the following command:

kubectl -n example_namespace get pod example_podname -o yaml | grep resources -A6

Note: Replace example-cluster with your cluster name, and example-podname with your Pod name.

Example output:

 containers:
        - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.1
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 600Mi

In the output, check the values for limits and requests. Make sure that the cpu and memory values are high enough for the node to keep the Pod. To view your Pod's requirements, run the following command:

kubectl describe pod podname -n namespace,

Note: Replace podname with your Pod name, and namespace with your namespace.

If the Pod requirements are higher than the node's abilities, then use the kubectl edit command to increase the limits and requests values for memory resources in the Cluster Autoscaler deployment. Or, create a new node group with a different instance type that fulfills the Pod's resource requirements. For more information about kubectl edit, see kubectl edit on the Kubernetes website.

Check the taint configuration for the node in the node group

To check whether you configured taints for the node, and the Pod can tolerate the taints, run the following command:

kubectl describe node example_nodename | grep taint -A2

Note: Replace example_nodename with your node name.

If you configured taints, then remove the taints defined on the node. For steps, see Your pod is in the Pending state on How do I troubleshoot the pod status in Amazon EKS? If the Pod can't tolerate taints, then define toleration on the Pod. For more information, see Taints and tolerations on the Kubernetes website.

Make sure that the node doesn't have scale down deactivated

To check for the status of scale-down-disable on your node, run the following command:

kubectl describe node example_nodename | grep scale-down-disable

Note: Replace example_nodename with your node name.

Example output:

cluster-autoscaler.kubernetes.io/scale-down-disabled: true

If scale-down-disable is set to true, then run the following command to remove the annotation for the node and allow it to scale down:

kubectl annotate node example_nodename cluster-autoscaler.kubernetes.io/scale-down-disabled-

Note: Replace example_nodename with your node name.

For more Cluster Autoscaler troubleshooting steps, see Frequently Asked Questions on the GitHub website.

Topics: Containers
Tags: Amazon Elastic Kubernetes Service
Language: English

AWS OFFICIALUpdated a year ago

No comments

Relevant content

EKS Cluster Autoscaler cannot scale nvidia.com/gpu nodes from zero
ssheng
asked 3 years ago
Cluster Autoscaler addon-resizer
rePost-User-9863136
asked a year ago
UnauthorizedOperation Error with Cluster Autoscaler in EC2 using VPC Endpoint
Sam
asked 2 years ago
Namespace error creating an EKS cluster
Systemgeek
asked 4 years ago
I am trying to create EKS cluster v 1.33 using AWS AL2023 AMI , EKS cluster gets created but no node joining the cluster
SushilKamboj
asked a year ago
How do I troubleshoot Pod scheduling issues with the Cluster Autoscaler in Amazon EKS?
AWS OFFICIALUpdated a year ago
How do I troubleshoot cluster creation failures in Amazon EMR on Amazon EKS?
AWS OFFICIALUpdated 2 years ago
How do I troubleshoot cluster creation errors in Amazon EKS?
AWS OFFICIALUpdated 2 years ago
How do I troubleshoot cluster scaling with Karpenter autoscaling in Amazon EKS?
AWS OFFICIALUpdated a year ago
Amazon EKS Security on High Alert with GuardDuty
EXPERT
Olawale Olaleye
published 2 years ago