Understanding Glue Service Limits and ResourceNumberLimitExceededException

This error indicates you've hit a specific AWS Glue resource limit related to the number of child tasks for a particular job group. The "ResourceNumberLimitExceededException" with the message "Number of ChildTask has reached the maximum limit of 10.0" suggests you've reached the maximum number of concurrent child tasks allowed for a specific job group.

AWS Glue has several service quotas that limit resource usage, including:

1. Maximum concurrent job runs per account (default is 30, not 2000)
2. Maximum concurrent job runs per job
3. Maximum task DPUs (Data Processing Units) per account
4. Maximum child tasks per job group (which appears to be 10 in your case)

To resolve this issue, you can try the following approaches:

1. Enable job queuing for your Glue jobs. This feature automatically queues job runs that exceed concurrent limits, putting them in a "WAITING" state until resources become available. You can enable this in the AWS Glue Studio console under the Job details tab by selecting "Enable job runs to be queued."

2. Implement a backoff strategy in your job submission process to avoid overwhelming the system with too many concurrent requests.

3. Optimize your job design to process multiple tables or tasks within a single job rather than running many separate jobs.

4. Request a quota increase through AWS Service Quotas if you consistently need more concurrent runs.

5. Monitor your resource usage through Amazon CloudWatch, which allows you to track metrics like ResourceUsage for various Glue resources.

The job queuing feature is particularly helpful as it will automatically handle concurrency limits without requiring you to manually retry failed jobs. Without queuing enabled, jobs will fail immediately with exceptions like the one you're seeing when limits are exceeded.
Sources
Glue job limit 2000 | AWS re:Post
Glue queue: Max concurrent runs exceeded | AWS re:Post
Monitoring AWS Glue resources - AWS Glue
Introducing job queuing to scale your AWS Glue workloads | AWS Big Data Blog

Hello,

ERORR:

+++++++ software.amazon.awssdk.services.gluejobexecutor.model.ResourceNumberLimitExceededException: Number of ChildTask has reached the maximum limit of 10.0 for groupId [...obfuscated...] (Service: GlueJobExecutor, Status Code: 400, Request ID: [...obfuscated...])

+++++++

This issue potentially happens when you run a Glue ETL job that uses a connection. AWS Glue creates elastic network interfaces (ENIs) in a VPC/private subnet to access your VPC data. These network interfaces then provide network connectivity for AWS Glue through your VPC. The number of ENIs depends on the number of data processing units (DPUs) selected for an AWS Glue ETL job. Specifically, you need as many ENIs as the number of DPUs configured for the job. AWS Glue DPU instances communicate with each other and with your database using ENIs. Each ENI is associated with an IP address, therefore, you need as many IP addresses as there are ENIs (DPUs). The number of available IP addresses depends on the number of host bits allowed when subnetting. Therefore, the solution to the issue is to use a larger subnet with enough free IP addresses.

In above case, you might find the below mentioned error in cloudwatch logs:

+++++++

The specified subnet does not have enough free addresses to satisfy the request. Please provide a connection with a subnet with ips available.

+++++++

RECOMMENDED SOLUTION:

+++++++

To overcome such errors :

1. If you are running multiple glue jobs create/use a subnet for the Glue connection with large range of IP addresses (more available IP addresses).
2. Create multiple glue connections with different subnets and use them in different sets of jobs as a workaround.

However, to confirm the above scenario, we would need to check the logs.

If you would like to discuss this further, please feel free to open a support case with AWS using the following link, and we would be happy to help.

Link to raise case: https://support.console.aws.amazon.com/support/home#/case/create

Thanks!