I want to troubleshoot cluster creation failures in Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS).
Resolution
Note:
Troubleshoot your cluster creation failures in Amazon EMR on Amazon EKS based on the error message that you receive.
An error occurred (ValidationException) when calling the CreateVirtualCluster operation
To designate a cluster namespace, you must create a Kubernetes role and bind the role to a Kubernetes user. Then, you must map the Kubernetes user with the AWSServiceRoleForAmazonEMRContainers service-linked role. If you don't have the appropriate permissions to access the namespace, then you receive the following error message:
"An error occurred (ValidationException) when calling the CreateVirtualCluster operation: Unauthorized to perform read namespace on (example-namespace)."
To resolve this issue, use Amazon EKS access entries.
Note: The Amazon EMR integration with Amazon EKS CAM is supported only for new Amazon EMR on Amazon EKS virtual clusters. You can't migrate current virtual clusters to use this integration. To use an access entry, run AWS CLI version 2.15.3 or later. Also, make sure that your Amazon EKS cluster version is 1.23 or later.
Complete the following steps:
- To confirm that your Amazon EKS cluster's authenticationMode is set to API_AND_CONFIG_MAP, run the describe-cluster command:
aws eks describe-cluster -name example-cluster
Note: Replace example-cluster with your cluster name.
- (Optional) To set authenticationMode to API_AND_CONFIG_MAP, run the update-cluster-config command:
aws eks update-cluster-config
-name example-cluster
-access-config authenticationMode=API_AND_CONFIG_MAP
Note: Replace example-cluster with your cluster name.
- Add the following permissions to the AWS Identity and Access Management (IAM) role that you use to run the CreateVirtualCluster and DeleteVirtualCluster API operations:
{
"Effect": "Allow",
"Action": [
"eks:CreateAccessEntry"
],
"Resource": "arn:<AWS_PARTITION>:eks:<AWS_REGION>:<AWS_ACCOUNT_ID>:cluster/<EKS_CLUSTER_NAME>"
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeAccessEntry",
"eks:DeleteAccessEntry",
"eks:ListAssociatedAccessPolicies",
"eks:AssociateAccessPolicy",
"eks:DisassociateAccessPolicy"
],
"Resource": "arn:<AWS_PARTITION>:eks:<AWS_REGION>:<AWS_ACCOUNT_ID>:access-entry/<EKS_CLUSTER_NAME>/role/<AWS_ACCOUNT_ID>/AWSServiceRoleForAmazonEMRContainers/*"
}
User: arn:aws:iam::example-account-id:user/example-username is not authorized to perform
When you're access is denied, you receive an error message similar to the following one:
"User: arn:aws:iam::example-account-id:user/example-username is not authorized to perform: example-action on resource: example-resource because example-reason."
When a policy denies access because of a Deny statement, the phrase with an explicit deny in a type policy is in the error message. When the policy denies access without a Deny statement, the phrase because no type policy allows the action is in the error message.
To submit and manage jobs on virtual clusters, create an IAM policy that allows you to create, list, describe, and delete virtual clusters. Also, to run Amazon EMR Spark jobs in Amazon EKS cluster namespaces, you can automate the AuthN and AuthZ policy configurations. For more information, see Permissions for managing virtual clusters.
To manage virtual clusters in your AWS account, create an IAM policy with the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole"
],
"Resource": "",
"Condition": {
"StringLike": {
"iam:AWSServiceName": "emr-containers.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": [
"emr-containers:CreateVirtualCluster",
"emr-containers:ListVirtualClusters",
"emr-containers:DescribeVirtualCluster",
"emr-containers:DeleteVirtualCluster"
],
"Resource": ""
}
]
}
Related information
How do I troubleshoot cluster creation errors in Amazon EKS?
How do I troubleshoot eksctl issues with Amazon EKS clusters and node groups?
Troubleshoot access denied error messages