Glue Role got S3 Access Denied when writing to underlying S3 data in Lake Formation

0

Hello, I have an ETL workflow and keep getting S3 Access Denied when the Glue Role write a data frame to a destination S3 bucket (registered in Lake Formation). Here is my current setup. Does Lake Formation vend credentials to the Glue Role for writing permission to S3? How to let the Glue Role write to S3 via Lake Formation?

  • Registered a S3 bucket to Lake Formation
  • Grant Data Location to the Glue Role
  • Grant Lake Formation Permissions (ALL) to the Glue Role
  • Attach AWSGlueServiceRole policy to the Glue Role
  • Attach an inline policy below to the Glue Role
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "iam:GetRole",
                "iam:PassRole",
                "lakeformation:GetDataAccess",
                "lakeformation:GrantPermissions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::amazon-reviews-pds",
                "arn:aws:s3:::amazon-reviews-pds/*"
            ],
            "Effect": "Allow"
        }
    ]
}
hai
asked a year ago612 views
2 Answers
0
Accepted Answer

Eventually, I found out that I use incorrect writing method in this case

  • Correct method: glueContext.write_dynamic_frame.from_catalog
  • Incorrect method: glueContext.write_dynamic_frame.from_option

Quoted from docs

Writes a DynamicFrame using the specified catalog database and table name.

docs

hai
answered a year ago
0

Once you register a location to be managed by LakeFormation, you should not use s3 permissions to handle it since it.
Now any client API trying to access that table needs to be LakeFormation away in order to be able to use the delegated access permissions, for instance DynamicFrame is able write into such table but not if you write directly to s3 or you use DataFrame.

profile pictureAWS
EXPERT
answered a year ago
  • The amazon-reviews-pds bucket is data source which is a publicly available. My lake (destination bucket) permissions not written in the policy and grant permissions via Lake Formation.

    Actually, if both IAM and Lake Formation grants permission, the the Glue Role is still able to write to S3, but it is not best practice.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions