InstanceFleets and Autoscaling

Question

We currently have an EMR cluster with the next configuration.

- Master instanceFleet ( 1 node r5a.xlarge on demand )
- Core instanceFleet ( 32 on demand units from multiple instance types )
- Task instanceFleet ( 24 spot units from multiple instance types with a price capacity allocation )

We'd like to change the cluster to a managed autoscaling but we don't fully understand the maths to calculate the 4 managed scaling parameters. We only want to scale the task instance fleet as the best practices tell to keep constant the core.

The number of units we want to have in task is a minimum of 2 and a maximum of 48.

1. Minimum capacity units: 1 Master + 32 core + 2 task = 35 ?
2. Maximum capacity units: 1 Master + 32 core + 48 task = 81 ?
3. Maximum core capacity units: 32 ?
4. Maximum on demand capacity units? 1 + 32 core = 33 ?

Are out maths correct?

Answer

An EMR cluster can only have one instance fleet per type i.e. one primary fleet, one core fleet, and one task fleet.

User defines capacity not instance count while configuring an instance fleet. Each instance type has a weight associated with it which means how many capacity units equate to a single instance of that type.

## Scale-out
**Given:**
    Cluster: My Cluster, id: j-ZVUZNG4PJR
    InstanceFleets:
        Core Instance Fleet, id if-33LN8PHHHF4NS

```
{
  "policyName": "ScaleOut_CoreFleet_MyCluster_if-33LN8PHHHF4NS",
  "serviceNamespace": "elasticmapreduce",
  "resourceId": "instanceFleet/j-ZVUZNG4PJR/if-33LN8PHHHF4NS",
  "scalableDimension": "elasticmapreduce:instancefleet:CoreFleetSpotCapacity",
  "policyType": "SimpleScaling", 
  "simpleScalingPolicyConfiguration": {
    "adjustmentType": "ChangeInCapacity",
    "scalingAdjustment": "1",
    "300": "number", // In number of seconds
  }
}
```

## Scale-in
**Given:**
    Cluster: My Cluster, id: j-ZVUZNG4PJR
    InstanceFleets:
        Core Instance Fleet, id if-33LN8PHHHF4NS

```
{
  "policyName": "ScaleIn_CoreFleet_MyCluster_if-33LN8PHHHF4NS",
  "serviceNamespace": "elasticmapreduce",
  "resourceId": "instanceGroup/j-ZVUZNG4PJR/if-33LN8PHHHF4NS",
  "scalableDimension": "elasticmapreduce:instancefleet:CoreFleetOnDemandCapacity",
  "policyType": "SimpleScaling", 
  "simpleScalingPolicyConfiguration": {
    "adjustmentType": "ChangeInCapacity",
    "scalingAdjustment": "-1",
    "300": "number", // In number of seconds
  }
}
```

InstanceFleets and Autoscaling

Scale-out

Scale-in

Relevant content