How do I resolve the error "Failed to start the job flow due to an internal error" in Amazon EMR?

Resolution

To resolve the error Failed to start the job flow due to an internal error in Amazon EMR, launch the cluster again. If the error still appears, then complete the following steps.

Check the permissions for the Amazon EMR service role

Make sure that the EMR_DefaultRole_V2 Amazon EMR service role has permissions to use the specified AWS Key Management Service (AWS KMS) keys. To launch Amazon EMR clusters, the Amazon EMR service role must have the following permissions:

{  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "EmrDiskEncryptionPolicy",
    "Effect": "Allow",
    "Action": [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:CreateGrant",
      "kms:GenerateDataKeyWithoutPlaintext",
      "kms:DescribeKey"
    ],
    "Resource": [
      "arn:aws:kms:us-west-2:<account-id>:key/<key-id>"
    ]
  }]
}

If the Amazon EMR cluster instances fail, then you might receive Connection timed out errors.

To troubleshoot Connection timed out errors, complete the following steps:

1. Open the Amazon Elastic Compute Cloud (Amazon EC2) console.
2. Select the terminated Amazon EC2 node.
   Note: The terminated node is available for 1 to 2 hours on the Amazon EC2 console.
3. On the Actions dropdown list, choose Monitor.
4. Choose Troubleshoot, and then choose Get system log.
5. Review your system log to determine the issue.

Check your Amazon VPC subnet routes

Make sure that your Amazon Virtual Private Cloud (Amazon VPC) subnet routes are correctly configured for the data source that your cluster uses. For more information, see Set up a VPC to host Amazon EMR clusters.

Check your security groups

Make sure that the primary, core, and task security groups are correctly configured for the subnet. For more information, see Working with Amazon EMR-managed security groups. Also, you must allow all required actions in the default Amazon EMR roles and the instance profile role. After you configure the Amazon VPC subnet routes, security groups, and roles, launch a new cluster.

Related information

Configure networking in a VPC for Amazon EMR