How do I resolve Amazon S3 AccessDenied errors in Amazon SageMaker training jobs?

Short description

AccessDenied errors occur when your AWS Identity and Access Management (IAM) policy doesn't allow one or more of the following Amazon Simple Storage Service (Amazon S3) actions:

• s3:ListBucket
• s3:GetObject
• s3:PutObject

The permissions that you need depend on the SageMaker API that you're calling. For example, the only Amazon S3 action that the CreateModel API requires is s3:GetObject. However, the CreateTrainingJob API requires s3:GetObject, s3:PutObject, and s3:ListObject. For more information about the permissions that are required for each API, see How to use SageMaker execution roles.

Resolution

AccessDenied errors occur in the following scenarios.

Encrypted input bucket

If the data in the S3 bucket is encrypted with AWS Key Management Service (AWS KMS), then check these permissions:

• Be sure that the IAM policy that's attached to the execution role allows the kms:encrypt and kms:decrypt actions. For more information, see How to use SageMaker execution roles.
• Be sure that the AWS KMS key policy grants access to the IAM role. For more information, see Key policies in AWS KMS.
• If you use an AWS KMS key for the machine learning (ML) storage volume in the resource configuration of your job, then the IAM policy must allow kms:CreateGrant action. For more information, see Grants in AWS KMS. For more information about how to encrypt the ML storage volume, see Protect data at rest using encryption.
• When you use the Python SDK and implement an abstraction of the estimator.EstimatorBase class, pass the output_kms_key and volume_kms_key parameters through kwargs keyword arguments. Do this regardless of their documented presence in the inheriting class. For more information, see Estimators.

Permissions boundaries

If you define permissions boundaries for the execution role, then SageMaker can run only the actions that are allowed by both the IAM policy and the permissions boundaries. Be sure that the IAM policy and the permissions boundaries allow the required Amazon S3 actions.

Bucket policies

If the input bucket uses a bucket policy, then be sure that the bucket policy allows the execution role to perform the required Amazon S3 actions. For more information about bucket policies, see Policies and permissions in Amazon S3.

Here's an example of a bucket policy that denies access to the SageMaker execution role and causes an AccessDenied error.

{
  "Version": "2012-10-17",
  "Id": "ExamplePolicy01",
  "Statement": [
    {
      "Sid": "ExampleStatement01",
      "Effect": "Deny",
      "Principal": {
        "AWS": "arn:aws:iam::Account-ID:role/SageMakerExecutionRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::awsdoc-example-bucket/*",
        "arn:aws:s3:::awsdoc-example-bucket"
      ]
    }
  ]
}

AWS account Amazon S3 access across accounts

If a different AWS account owns the Amazon S3 data, then check for these permissions:

• Be sure that both accounts have access to the AWS KMS key. If you don't specify an AWS KMS key for the training job, then SageMaker defaults to an Amazon S3 server-side encryption key. A default Amazon S3 server-side encryption key can't be shared with or used by another AWS account.
• Be sure that the IAM policy for the SageMaker execution role and the S3 bucket policy have permissions across accounts.

For more information, see How can I deploy an Amazon SageMaker model to a different AWS account?

Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?