AWS Glue load into cross account S3

Question

Hi team,
I want to read tables from redshift using glue (Account-a) and perform some transformation and load into S3 (Account-b). Is there any way I can do this. I have added trust relationships in account-b and added bucket policy for the bucket so that glue can assume the role and load into S3.

Trust relationship:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "glue.amazonaws.com",
                "AWS": "arn:aws:iam:::root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

Bucket Policy:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGlueAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam:::root"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::test-bucket",
                "arn:aws:s3:::test-bucket/*"
            ]
        }
    ]
```

Glue Script:
```
bucket_name = "test-bucket"
key = "000.csv"
s3_client = boto3.client('s3')

df = glueContext.create_dynamic_frame.from_options(connection_type="redshift") // I am able to read table
df = df.toDF()
csv_buffer = StringIO()
df.toPandas().to_csv(csv_buffer, index=False)

s3_client.put_object(Bucket=bucket_name, Key=key, Body=csv_buffer.getvalue())
```

I am getting an error: Error Category: TIMEOUT_ERROR; ConnectTimeoutError: Connect timeout on endpoint URL

I did try to assume role using script and getting error: Error Category: TIMEOUT_ERROR; ConnectTimeoutError: Connect timeout on endpoint URL: "https://sts.amazonaws.com/"

Any help would be much appreciated

Elena Madrazo · Answer

Hello,
Thank you very much for your question. Based on the error you're encountering and the information provided, it seems that the issue is related to network connectivity or firewall rules preventing access to the AWS Security Token Service (STS) endpoint. Below are some steps and further information to troubleshoot and resolve the issue:
1. Check Network Connectivity: Ensure that your network has a stable internet connection and can reach the AWS STS endpoint (https://sts.amazonaws.com/). You can try pinging the endpoint from your local machine or EC2 instance to verify connectivity.

2. Check Firewall Rules: If you're running your script from an EC2 instance or behind a corporate firewall, ensure that the necessary outbound rules are configured to allow traffic to the AWS STS endpoint.

3. Check AWS Service Limits: Verify that you haven't reached any AWS service limits, such as the maximum number of concurrent requests to STS or the maximum session duration for assumed roles.

4. Check IAM Role and Policy Configuration: Double-check the IAM role and policy configurations in both accounts to ensure that the necessary permissions are granted correctly.

5. Check AWS Glue Job Timeout Settings: If you're running the AWS Glue job with a timeout setting, increase the timeout value to allow more time for the job to complete.

6. Check AWS Glue Job Logs: Review the AWS Glue job logs for any additional error messages or clues that could help identify the root cause of the issue.

If the issue persists after trying these steps, you may need to reach out to AWS Support for further assistance.
Furthermore, make sure to double-check the IAM role configurations, bucket policies, and network connectivity to ensure that the necessary permissions and access are granted correctly.
Further links:
https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html
https://repost.aws/knowledge-center/connect-timeout-endpoint-url-sagemaker

AWS Glue load into cross account S3

Relevant content