AWS EMR Serverless ServiceQuotaExceededException

Question

Getting this error when trying to run a simple spark job which reads a json file from s3 and prints the schema.

```
"com.amazonaws.emr.serverless.shaded.software.amazon.awssdk.services.emrserverlessresourcemanager.model.ServiceQuotaExceededException: Worker could not be allocated as the account has reached the service limit on the maximum vCPU it can use concurrently "
```

I am running just this one job with 1 driver.core and 1 executor.core and 1 executor instance. The ServiceQuota is set to concurrent use of 16vCPUs. Can someone explain how might I be hitting the Service Quota limit?

Answer

Hi Max,

My job is running with this config "--conf spark.executor.cores=1 --conf spark.executor.memory=2g --conf spark.driver.cores=1 --conf spark.driver.memory=2g --conf spark.executor.instances=1 " and the ServiceQuotaLimit is 16 vCPUs. I am not able to understand how this is adding up to 16. Need to understand that to calculate what limit I should request for.

Answer

You can read more about vCPU account limits here: https://aws.amazon.com/blogs/compute/preview-vcpu-based-instance-limits/

To request an increase - first determine how many vCPUs you need - then open a support case, and ask for a limit increase to the number of vCPUs that you need.  Follow the process discussed in the [EC2 Knowledge Center Article on vCPU Limit Increases](https://repost.aws/knowledge-center/ec2-on-demand-instance-vcpu-increase).

Answer

From the documentation, it seems that `spark.dynamicAllocation.enabled` is True by default, the `spark.dynamicAllocation.maxExecutors` default value is infinite (for version 6.10.0 and higher):
https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html so our job was creating a high number of workers. We are going to disable this option and see if we still hit the vCPU limits.

AWS EMR Serverless ServiceQuotaExceededException

Relevanter Inhalt