跳至内容

如何排查在 SageMaker AI Studio 中尝试运行计划笔记本作业时收到的错误?

2 分钟阅读
0

我想排查在 Amazon SageMaker AI Studio 中尝试运行计划笔记本作业时收到的错误。

解决方法

对 AccessDenied 错误进行故障排除

当计划的笔记本作业尝试运行时,您可能会由于以下原因收到 "AccessDenied" 错误:

  • 您没有所需的 AWS Identity and Access Management (IAM) 策略。
  • 您没有所需的 Amazon Virtual Private Cloud (Amazon VPC) 端点策略。
  • 您的资源标签异常。

IAM 策略问题

确保您的笔记本将以下策略附加到 IAM 角色,以允许建立基本信任关系:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sagemaker.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

验证您的 IAM 角色是否具有以下权限:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::*:role/*",
      "Condition": {
        "StringLike": {
          "iam:PassedToService": [
            "sagemaker.amazonaws.com",
            "events.amazonaws.com"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "events:TagResource",
        "events:DeleteRule",
        "events:PutTargets",
        "events:DescribeRule",
        "events:PutRule",
        "events:RemoveTargets",
        "events:DisableRule",
        "events:EnableRule"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:PutBucketVersioning",
        "s3:PutEncryptionConfiguration"
      ],
      "Resource": "arn:aws:s3:::sagemaker-automated-execution-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sagemaker:ListTags"
      ],
      "Resource": [
        "arn:aws:sagemaker:*:*:user-profile/*",
        "arn:aws:sagemaker:*:*:space/*",
        "arn:aws:sagemaker:*:*:training-job/*",
        "arn:aws:sagemaker:*:*:pipeline/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags"
      ],
      "Resource": [
        "arn:aws:sagemaker:*:*:training-job/*",
        "arn:aws:sagemaker:*:*:pipeline/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:CreateNetworkInterfacePermission",
        "ec2:CreateVpcEndpoint",
        "ec2:DeleteNetworkInterface",
        "ec2:DeleteNetworkInterfacePermission",
        "ec2:DescribeDhcpOptions",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcEndpoints",
        "ec2:DescribeVpcs",
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetEncryptionConfiguration",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject",
        "sagemaker:DescribeDomain",
        "sagemaker:DescribeUserProfile",
        "sagemaker:DescribeSpace",
        "sagemaker:DescribeStudioLifecycleConfig",
        "sagemaker:DescribeImageVersion",
        "sagemaker:DescribeAppImageConfig",
        "sagemaker:CreateTrainingJob",
        "sagemaker:DescribeTrainingJob",
        "sagemaker:StopTrainingJob",
        "sagemaker:Search",
        "sagemaker:CreatePipeline",
        "sagemaker:DescribePipeline",
        "sagemaker:DeletePipeline",
        "sagemaker:StartPipelineExecution"
      ],
      "Resource": "*"
    }
  ]
}

有关详细信息,请参阅 SageMaker AI Notebook 的 AWS 托管式策略

VPC 端点问题

如果通过 Amazon VPC 端点启动笔记本作业,请检查该端点的配置和策略。确保完成所需步骤并遵循相关 AWS 服务端点的最佳实践:

对于 Amazon S3 VPC 端点,您可能会收到与限制为单个 AWS 账户的端点相关的错误。例如,以下策略限制对 ID 为 111122223333 的账户的访问权限:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSpecificAccountsPermission",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "s3:ResourceAccount": "111122223333"
        }
      }
    }
  ]
}

要解决此问题,您还需要允许用户执行以下 S3 存储桶访问操作:

{
  "Action": [
    "s3:*"
  ],
  "Resource": [
    "arn:aws:s3:::sagemakerheadlessexecution-prod-*",
    "arn:aws:s3:::sagemakerheadlessexecution-prod-*/*"
  ],
  "Effect": "Allow",
  "Sid": "SCTASK14554266"
}

资源标签异常

确保您的 IAM 策略具有以下权限:

{
  "Effect": "Allow",
  "Action": [
    "events:TagResource",
    "events:DeleteRule",
    "events:PutTargets",
    "events:DescribeRule",
    "events:PutRule",
    "events:RemoveTargets",
    "events:DisableRule",
    "events:EnableRule"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
    }
  }
}

对 UI 错误进行故障排除

当您尝试创建描述更新停止删除笔记本作业时,可能会收到 UI 错误。当您使用作业定义(计划作业)时,也可能会收到此错误。要进行故障排除,请查看 UI 中显示的错误消息。此消息可能包含解决问题的指示或建议的操作。

如果无法解决错误,请完成以下步骤:

  1. 获取错误的屏幕截图,并将其另存为图像文件。
  2. 创建 HTTP 存档 (HAR) 文件,用于在出现 UI 错误时捕获网络流量。
  3. 打开 SageMaker AI Studio Jupyter 服务器终端。依次选择文件、新建、终端
  4. 检查 /var/log/apps/app_container.log 中的日志,查看 UI 出现错误时是否有异常、错误或警告。
  5. 联系 AWS Support。在您的请求中,附上错误屏幕截图、app_container.log 和 HAR 文件。
AWS 官方已更新 8 个月前