How do I troubleshoot Pod scheduling issues with the Cluster Autoscaler in Amazon EKS?

Resolution

If you activated Cluster Autoscaler and your Pod is stuck in the Pending state, then Cluster Autoscaler might not add new nodes to unscheduled Pods.

Check the state of your Pod

To check the state of your Pod, run the following kubectl get pods command:

kubectl get pods -n kube-system -l app=cluster-autoscaler

If the Pod is in the Pending state or if your Pod's container is in the Waiting state, then troubleshoot the Pod's status.

If your Pod is in the CrashLoopBackOff state, then run the following kubectl describe pod command:

kubectl describe pod cluster-autoscaler-####-#### -n kube-system

Note: Replace cluster-autoscaler-####-#### with your Cluster Autoscaler Pod.

If the command's output shows "OOMKilled" with the 137 exit code, then increase your memory resource limits and requests for your cluster-autoscaler deployment. For more information, see Specify a memory request and a memory limit on the Kubernetes website.

To check the Pod logs, run the following kubectl logs command:

kubectl logs -f -n kube-system -l app=cluster-autoscaler

The logs can provide troubleshooting information. For example, the Pod log might show the following "AccessDenied" error message:

"Failed to create AWS Manager: AccessDenied: User: arn:aws:sts::444455556666:assumed-role/your-role is not authorized to perform: your-action because no identity-based policy allows the your-action action"

To resolve AWS Identity and Access Management (IAM) permissions issues, attach the correct IAM policy to the Pod. For more information, see IAM policy on the GitHub website.

Or, the log might show the following error message:

"Failed to create AWS Manager: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials caused by: RequestError: send request failed caused by: Post https://sts.region.amazonaws.com/: : dial tcp: i/o timeout"

If you're experiencing network connectivity issues, then verify that your worker node subnet has a route to the following AWS service endpoints:

• ec2.region.amazonaws.com
• autoscaling.region.amazonaws.com
• sts.region.amazonaws.com

Also, verify that your subnet's network access control list (network ACL) or worker node security group doesn't block traffic to the preceding endpoints.

If your cluster is private, then take the following actions:

• Add the preceding endpoints to your virtual private cloud (VPC).
• Configure the security group for endpoints to allow traffic on port 443 from a worker node security group.

Identify why your Cluster Autoscaler can't scale out the Auto Scaling group

To check whether your Pod contains a scheduling rule, such as affinity or anti-affinity, run the following kubectl describe pod command:

kubectl describe pod pending-pod-name -n pending-pod-namespace

Note: Replace pending-pod-name with the name of your pending Pod and pending-pod-namespace with your pending Pod's namespace. For more information, see Affinity and anti-affinity on the Kubernetes website.

In the command's output, check the Events section to determine why your Pod is stuck in the Pending state. For example, you might identify an issue with your node group labels.

To make sure that your node groups work, take the following actions:

• Configure your node groups with labels that match the nodeSelector and requiredDuringSchedulingIgnoredDuringExecution settings in your Pod's nodeAffinity specification. For more information, see Node labels on the Kubernetes website. The correct labels allow Cluster Autoscaler to identify eligible nodes for scaling operations.
• Make sure that you correctly defined your Pod scheduling rules. For more information, see Assigning Pods to nodes on the Kubernetes website. When Pods use nodeSelector or requiredDuringSchedulingIgnoredDuringExecution, the Cluster Autoscaler considers only scaling node groups that match the scheduling requirements for these settings.

To check the Cluster Autoscaler configuration, run the following kubectl get deployment command:

kubectl get deployment cluster-autoscaler -n kube-system -o yaml

If the Cluster Autoscaler has the correct node-group-auto-discovery argument, then the command's output shows the following configuration:

command:
  - ./cluster-autoscaler
  - --v=4
  - --stderrthreshold=info
  - --cloud-provider=aws
  - --skip-nodes-with-local-storage=false
  - --expander=least-waste
  - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/your-cluster-name

Check whether the Auto Scaling group has the correct tags.

Cluster Autoscaler requires the following tags to discover the Auto Scaling group that's associated with the node group:

Tag 1:
key: k8s.io/cluster-autoscaler/your-cluster-name
value: owned

Tag 2:
key: k8s.io/cluster-autoscaler/enabled
value: true

To check whether the node group reached the maximum number of nodes, run the following describe-nodegroup command:

aws eks describe-nodegroup --cluster-name your-cluster-name --nodegroup-name your-nodegroup-name

Note: Replace your-cluster-name with the name of your cluster and your-nodegroup-name with the name of your node group.

If the node group reached the maximum number of nodes, then update the node group to increase the maximum node count. Then, the new nodes can schedule the new Pods.

Check whether the Amazon Elastic Compute Cloud (Amazon EC2) instance that your Auto Scaling group launched can join your cluster. If the instance can't join your cluster, then troubleshoot your worker nodes.

To check the Pod resource request, run the following kubectl get pod command:

kubectl get pod pending-pod-name -n pending-pod-namespace -o yaml | grep resources -A6

Note: Replace pending-pod-name with the name of your pending Pod and pending-pod-namespace with the name of your pending Pod's namespace.

Check whether the current node instance type fulfills the Pod resource request, and modify the Pod resource request if needed. Or, create a new node group with a modified instance type. For more information about Pod resource requests, see Resource management for Pods and containers on the Kubernetes website.

To view the taints for a node, run the following kubectl describe node command:

kubectl describe node your-node-name

Note: Replace your-node-name with the name of your node.

Make sure that your Pods tolerate the taints that you added to the node. If the node doesn't require the taints, then remove the taints from the node. For more information, see Taints and tolerations on the Kubernetes website.

Related information

Frequently asked questions on the GitHub website