By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EKS scheduler bin-packing

2

Hello, we are executing batch jobs on EKS as pods and we are facing issue with workloads are spread across large number of nodes after scale-up with low requests. Running jobs can't be migrated to other node so autoscaler ignores them and it prevents scale-down.

It might be helpful to binpack these job pods across the nodes similar as described https://alibaba-cloud.medium.com/the-burgeoning-kubernetes-scheduling-system-part-3-binpack-scheduling-that-supports-batch-jobs-372b4704722 or https://kubernetes.io/docs/concepts/scheduling-eviction/resource-bin-packing/#enabling-bin-packing-using-requestedtocapacityratio Is it possible to activate bin-pack scheduler on EKS? Or which approach would you recommend for this situation? thanks Martin

Topics
ContainersServerlessCompute
Tags
Amazon Elastic Kubernetes ServiceJob Scheduling
Language
English
MartinHalamicek
asked 2 years ago590 views
1 Answer
0

You might want to try using https://karpenter.sh/ for autoscaling and see if it improves your utilization. It can also consolidate pods to less nodes, but that might not always be appropriate for running jobs

AWS
dov
answered a year ago
  • Leon
    5 months ago

    We are already using Karpenter and it does not solve the problem but instead makes it worse. Karpenter creates much bigger nodes during scale-up, than cluster autoscaler, which are much more underutilized after the load goes away.

    To fix the problem we need to be able to adapt the scheduling policy of the kubernetes scheduler to use its bin packing capability. Then new pods would not be spread across all nearly empty nodes but be bin packed on just some nodes which result in some empty nodes which can then be removed by karpenter.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content