How do I resolve Amazon S3 AccessDenied errors in SageMaker AI training jobs?

Short description

An AccessDenied error might occur if your AWS Identity and Access Management (IAM) policy doesn't allow the following Amazon Simple Storage Service (Amazon S3) actions:

• s3:ListBucket
• s3:GetObject
• s3:PutObject

Your SageMaker AI API call determines the required permissions. For example, the CreateModel API requires only s3:GetObject, but the CreateTrainingJob API requires s3:GetObject, s3:PutObject, and s3:ListObject.

Resolution

To resolve an AccessDenied error, take the following actions based on your scenario.

Encrypted input bucket

If you encrypted data in the S3 bucket with AWS Key Management Service (AWS KMS), then check the permissions. The IAM policy that's attached to the execution role must allow the kms:encrypt and kms:decrypt actions. Check that the AWS KMS key policy grants access to the execution role.

You might use an AWS KMS key for the machine learning (ML) storage volume in the resource configuration of your job. If you use an AWS KMS key, then the IAM policy must allow the kms:CreateGrant action. For more information, see Grants in AWS KMS.

Note: It's a best practice to encrypt the ML storage volume.

When you use the Python SDK and implement an abstraction of the estimator.EstimatorBase class, pass the output_kms_key and volume_kms_key parameters through kwargs keyword arguments. Take this action regardless of the documented presence of the output_kms_key and volume_kms_key parameters in the inheriting class. For more information, see Estimators on the Amazon SageMaker Python SDK website.

Permissions boundaries

If you define permissions boundaries for the execution role, then SageMaker AI only runs actions that are allowed by both the IAM policy and the permissions boundaries. Make sure that the IAM policy and the permissions boundaries allow the required Amazon S3 actions.

Bucket policies

If the input bucket uses a bucket policy, then the bucket policy must allow the execution role to perform the required Amazon S3 actions.

The following is an example of a bucket policy that denies access to the SageMaker AI execution role and causes an AccessDenied error:

{  
  "Version": "2012-10-17",
  "Id": "ExamplePolicy01",
  "Statement": [
    {
      "Sid": "ExampleStatement01",
      "Effect": "Deny",
      "Principal": {
        "AWS": "arn:aws:iam::Account-ID:role/SageMakerExecutionRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::awsdoc-example-bucket/*",
        "arn:aws:s3:::awsdoc-example-bucket"
      ]
    }
  ]
}

Note: To allow access to the execution role, replace Deny with Allow for the Effect value.

Cross AWS account access

If a different AWS account owns the Amazon S3 data, then check that both accounts have access to the AWS KMS key. If you don't specify an AWS KMS key for the training job, then SageMaker AI defaults to an Amazon S3 server-side encryption key. Another account can't share or use a default Amazon S3 server-side encryption key.

Make sure that the IAM policy for the SageMaker AI execution role and the S3 bucket policy have permissions across accounts.

For more information, see How can I deploy an Amazon SageMaker AI model to a different AWS account?

Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?

How to use SageMaker AI execution roles