Studio Workspace can't see my runnning EMR EC2 cluster to attach to

Question

Let me know if this is something AWS EMR Studio does:
1. in Databricks community edition, and in Google Collab, one can fire up a simple Jupyter notrebook with an automatically started cluster (small default) to run PySpark commands
2. I'm trying to do something like this with AWS and have tried these 2 approaches
     a. EMR serverless, but it keeps timing out so ..
     b. instead I start up a minimal EMR EC2 cluster, i can see VPC and Subnet as defaults and it is ready and waiting

3. But when I try to open Studio / Workspace, I can't get it to recognize or see the EMR EC2 cluster to attach to

Any ideas of what I am missing?

Answer

Hello,

Can you check if your IAM user has enough permissions to describe the cluster in the EMR studio execution arn. Additionally, a workspace in Studio that has IAM Identity Center trusted identity propagation enabled can only attach to an EMR cluster with a security configuration that has Identity Center enabled.

On the other hand, you can also create EMR on EC2 cluster in the Notebook by selecting the Compute and expand the advanced configuration option as shown below, To create clusters, your Studio administrator must first give you permission using a session policy. For more information, please refer this [document](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-use-clusters.html#emr-studio-create-cluster).

![Enter image description here](/media/postImages/original/IMFWU0__klRlqVMvAogMm1yg)

Studio Workspace can't see my runnning EMR EC2 cluster to attach to

관련 콘텐츠