How can I resolve the AWS Glue error "The specified subnet does not have enough free addresses to satisfy the request"?

Short description

When you run an AWS Glue extract, transform, or load job, you might see one of the following errors:

"The specified subnet does not have enough free addresses to satisfy the request. (Service: AmazonEC2; Status Code: 400; Error Code: InsufficientFreeAddressesInSubnet; Request ID: my_request_id)"

"An error occurred while calling o70.getDynamicFrame. Job 0 cancelled because SparkContext was shut down caused by threshold for executors failed after launch reached"

These errors occur when there aren't enough IP addresses available for the AWS Glue job. There are two common reasons why there aren't enough IP addresses:

• If you run a job in a virtual private cloud (VPC) subnet, then AWS Glue sets up elastic network interfaces. Each elastic network interface establishes a secure connection to other resources in the VPC and gets a private IP address. If the elastic network interfaces aren't released as expected, then there might not be enough IP addresses available for the job. To resolve the error, confirm how many data processing units (DPUs) the job used and reduce the number of DPUs.
• Multiple AWS services on the same subnet. These services might use the subnet's available IP addresses. To resolve the error, use a different subnet with more available IP addresses for the AWS Glue job.

Resolution

Reduce the number of DPUs for the job

Determine how many DPUs were used by a job

To check how many DPUs were used by the job, first let the job complete, and then do the following:

1. Open the AWS Glue console.
2. On the navigation pane, choose Jobs.
3. Choose the job, and then choose the History tab. The Maximum capacity column shows the number of DPUs used for the job.

Reduce the number of DPUs

Decide how many DPUs to remove from the job. Note that the number of DPUs doesn't equal the number of elastic network interfaces. One elastic network interface is always attached to each worker.

The following additional elastic network interfaces are also required for each job:

• Standard worker type (1 DPU per worker): one additional elastic network interface required
• G1.X worker type (1 DPU per worker): one additional elastic network interface required
• G2.X worker type (2 DPUs): one additional elastic network interface required

For example, if you run a job with 20 workers on a G.2x, calculate the number of IP addresses as follows:

• With AWS Glue 0.9/1.0: 41 DPU = 20 workers (executors) + 1 driver + 1 task runner = 22 IP addresses
• With AWS Glue 2.0/3.0/4.0: 40 DPU = 19 Workers (executors) + 1 driver = 20 IP addresses (no task runner in AWS Glue 2.0/3.0/4.0)

To reduce the number of DPUs, do the following:

1. On the navigation pane, choose Jobs.
2. Choose the Action dropdown list, and then choose Edit job.
3. Expand the Security configuration, script libraries, and job parameters (optional) list.
4. In the Maximum capacity field, enter a lower number. This field sets the maximum number of DPUs that the job can use.
5. Save your changes, and then run the job again.

Delete unused elastic network interfaces

To check the number of available IP addresses in the subnet, do the following:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select a connection.
4. In the Action dropdown list, choose View details. Note the subnet.
5. Open the Amazon VPC console.
6. In the navigation pane, choose Subnets.
7. In the Subnet dropdown list, choose the subnet that the AWS Glue connection uses, as noted in step 4.
8. On the Description tab, find the Available IPv4 Addresses field. This field shows how many IP addresses are available in the subnet.
9. Confirm that the number of available IP addresses is more than the number of IP addresses required for the AWS Glue job.

If the number of available IP addresses is less than what's required for your AWS Glue job to run, then delete the unused elastic network interfaces.

Create and use a subnet with more available IP addresses

To create a new subnet, do the following:

1. Create a new subnet in your VPC.
   Note: You can create a new subnet that uses the VPC's original CIDR blocks. Or, you can add additional CIDR blocks to your VPC to use with the new subnet.
2. Review the route tables and access control list (ACL) rules associated with the old subnet to be sure that the new subnet routes traffic the same way. For example, suppose your previous subnet had a default route configured to an internet gateway. You must make sure that your new subnet has a similar default route.

To modify the AWS Glue connection to use the new subnet, do the following:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select the connection used by your AWS Glue job.
4. In the Action dropdown list, choose Edit connection.
5. On the Set up your connection's properties page, choose Next.
6. On the Set up access to your data store page, in the Subnet dropdown list, choose the new subnet.
7. Choose Next, and then choose Finish.
8. Run the job again.

Related information

Defining job properties for Spark jobs

Setting up a VPC to connect to JDBC data stores