EMR Cluster Auto Termination Not Working

Question

I want my EMR cluster to be terminated automatically post an idle time.
I have configured 'Automatically terminate cluster after idle time' and set the idle time as '5 minutes' .
In my cluster i have selected spark alone. and manually created the console through console.

post creation of EMR cluster i did not perform any activity in the cluster but still the cluster is not getting terminated automatically.
I have waited event 6 hours , the termination did not happen.

As per the documentation - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-auto-termination-policy.html
I have added below permission to my EMR cluster IAM role but still not change 
{
  "Version": "2012-10-17",
  "Statement": {
      "Sid": "AllowAutoTerminationPolicyActions",
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:PutAutoTerminationPolicy",
        "elasticmapreduce:GetAutoTerminationPolicy",
        "elasticmapreduce:RemoveAutoTerminationPolicy"
      ],
      "Resource": ""
    }
}

Kindly help me to figure it out

Answer

Hello,

As you might know that “AutoTerminationIsClusterIdle” CloudWatch metric Indicates whether the cluster qualifies for AutoTermination or not. A value of 0 indicates that the cluster is in active use by one of the following components:
——————————————————————————————————
1. An active YARN application 
2. HDFS utilization greater than 10%
3. Active EMR notebook or EMR Studio connections
4. An on-cluster UI, such as the Spark History Server

For 1. You can check "AppsRunning", "AppSubmitted" or "ContainerPendingRatio" metric.

For 2. You can check "HDFSUtilization" metric.

For 3. You can check "TotalNotebookKernels" metric

There could be other reasons as well that you can check in the metric collector log in the primary node. I recommend to check all the considerations given in this document to verify them all are satisfied in your cluster. For an instance, below,

To use auto-termination, the metrics-collector process must be able to connect to the public API endpoint for auto-termination in API Gateway. If you use a private DNS name with Amazon Virtual Private Cloud, auto-termination won't function properly. To ensure that auto-termination works, we recommend that you take one of the following actions:

-Remove the API Gateway interface VPC endpoint from your Amazon VPC.
-Follow the instructions in Why do I get an HTTP 403 Forbidden error when connecting to my API Gateway APIs from a VPC?[1] to disable the private DNS name setting.
-Launch your cluster in a private subnet instead. For more information, see the topic on Private subnets. Please refer below docs for more details.

[+] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-auto-termination-policy.html#emr-auto-termination-considerations

[+] https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_ViewingMetrics.html

EMR Cluster Auto Termination Not Working

相关内容