Why aren't my Spot Instances provisioned during an EMR cluster resize?

Resolution

Amazon Elastic Compute Cloud (Amazon EC2) might interrupt your Spot Instance at any time for the following reasons:

• Lack of Spot capacity.
• Amazon EC2 can't meet the request constraints.
• The Spot Price is higher than the designated maximum price.
• Your Spot account quota is exhausted. To resolve this issue, you can request a service quota increase.

For more information, see Why did Amazon EC2 interrupt my Spot Instance?

Note: It's a best practice to use Spot Instances only for workloads that are stateless, fault-tolerant, and flexible enough to withstand interruptions.

Spot Instances and On-Demand Instances also might not be provisioned because the bootstrap scripts are modified or contain errors.

Check the logs for the bootstrap script at /emr/instance-controller/log/bootstrap-actions or s3://cluster_id/node-failed/bootstrap-actions/stderr.gz.

For example, the following bootstrap action log shows that bootstrap action 1 (emr_bootstrap_actions.sh) failed:

Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum
    Memory : 125 M RSS (444 MB VSZ)
    Started: Tue Jul 19 05:36:36 2022 - 00:03 ago
    State  : Running, pid: 7914
Error: Package: falcon-sensor-4.18.0-6403.amzn2.x86_64 (/falcon-sensor-4.18.0-6403.amzn2.x86_64)
           Requires: systemd

The preceding error causes all new replacement nodes terminate. The node also stops provisioning new replacement instances and the core node instance group goes into arrested with the following error message:

""state": "ARRESTED", "message": "Instance group ig-2JN5######## in Amazon EMR cluster j-37H4####### (emr-#####-spark-cluster) was arrested at for the following reason: Error provisioning instances.""

Correct the errors in the bootstrap script before you restart the cluster resize.

Related information

Spot Instance interruptions

Get the status of a Spot Instance request

Why is my Spot Instance terminating even though the maximum price is higher than the Spot price?