A SageMaker Pipeline I built using a Jupyter notebook in SageMaker Studio has a SageMaker Processing Job step. However, the step fails within 5 minutes with the following message: ClientError: Failed to download data. AccessDenied (403): User: arn:aws:sts::<destination-acc-id>:assumed-role/CTE-AmazonSageMakerStudio-ExecutionRole/SageMaker is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access. It's unable to copy data from another account's S3 bucket to Processing Job's volume.

I believe this is because within the pipeline session, Processing Job uses an AssumedRole to copy cross-account S3 data. If I run the step without the pipeline session context it works. Here's the Role that it uses without pipeline context: arn:aws:iam::<destination-acc-id>:role/CTE-AmazonSageMakerStudio-ExecutionRole

Is there a way to force the SageMaker PipelineSession to use the role that works?

source-acc: The account that has the data I'm trying to copy
destination-account: The account running the SageMaker Processing Job. The job tries to copy data to the local volume

Python3.9:

import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import PipelineSession
from sagemaker.tensorflow import TensorFlowProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.workflow.steps import ProcessingStep

data_version = "2024_01_06-1"
pipeline_sess = PipelineSession()

processor = TensorFlowProcessor(
    role=sagemaker.get_execution_role(),
    sagemaker_session=pipeline_sess,
    instance_count=1,
    instance_type="ml.c5.18xlarge",
    py_version="py39",
    framework_version="2.11.0",
    volume_size_in_gb=40,
    max_runtime_in_seconds=int(6*3600),
    env={"AWS_DEFAULT_REGION": "us-east-1"},
)
proc_input = ProcessingInput(
    source=f"s3://{source-bucket}/ClassificationExperiments/{data_version}/raw/meta",
    destination=f"/opt/ml/processing/input_data/meta",
    s3_data_type="S3Prefix",
    s3_input_mode="File",
    s3_data_distribution_type="FullyReplicated",
    s3_compression_type="None",
    app_managed=False,
)
proc_output_meta = ProcessingOutput(
    source="/opt/ml/processing/output_data/meta",
    destination=f"s3://{destination-acc-bucket}/ClassificationExperiments/{data_version}/raw/meta",
    s3_upload_mode="EndOfJob",
    app_managed=False,
)
step_y_data = ProcessingStep(name="YData",
                            step_args=processor.run(code="main.py",
                                                    source_dir="./LocalFolder",
                                                    inputs=[proc_input],
                                                    outputs=[proc_output_meta],
                                                    ),
)
pipeline = Pipeline(name="TestPipeline",
                    sagemaker_session=pipeline_sess,
                    steps=[step_y_data],
                   )
pipeline.upsert(role_arn=sagemaker.get_execution_role())
execution = pipeline.start()

Here's the Trust Relationship associated with arn:aws:iam::<destination-acc-id>:role/CTE-AmazonSageMakerStudio-ExecutionRole

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "sagemaker.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Here's the SSE-KMS Key Policy associated with the source bucket I'm trying to copy data from:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<source-acc-id>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<destination-acc-id>:role/CTE-AmazonSageMakerStudio-ExecutionRole"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

Here's the Bucket Policy associated with the source bucket I'm trying to copy data from:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Cross account read object S3 bucket policy",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<destination-acc-id>:role/CTE-AmazonSageMakerStudio-ExecutionRole"
            },
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::<source-bucket>",
                "arn:aws:s3:::<source-bucket>/*"
            ]
        },
        {
            "Sid": "Cross account read object S3 bucket policy",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<source-account-id>:root"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<source-bucket>",
                "arn:aws:s3:::<source-bucket>/*"
            ]
        }
    ]
}