Why do I get "not authorized to perform s3:DeleteObject" errors when running TRUNCATE commands in AWS Glue with Lake Formation hybrid access mode?

Short description

In Lake Formation hybrid access mode, credential vending only occurs when principals are explicitly opted-in. Without opt-in, Lake Formation intercepts S3 requests but doesn't provide temporary credentials, causing authorization failures even when IAM permissions are correctly configured. This article explains how to opt-in principals to enable credential vending and resolve TRUNCATE authorization errors.

Resolution

Understanding the root cause

When an S3 location is registered with Lake Formation in hybrid access mode, principals must be opted-in to use individual Lake Formation permissions and credential vending. However, the ALL permission has special bypass behavior that allows it to work without opt-in.

Lake Formation hybrid access mode supports two authorization paths:

1. IAM path (for non-opted-in principals): Uses IAM permissions only, but Lake Formation still intercepts S3 requests without providing credentials
2. Lake Formation path (for opted-in principals): Uses Lake Formation permissions with credential vending

When your Glue role is not opted-in, Lake Formation blocks S3 operations because the location is registered, but it doesn't provide the temporary credentials needed to access the data.

Authorization flow diagram:

                         Glue Job Executes TRUNCATE
                                   |
                                   v
              Is S3 location registered with Lake Formation?
                                   |
                              YES (Hybrid)
                                   |
                                   v
         Does table have IAMAllowedPrincipals group permissions?
                                   |
                   +---------------+---------------+
                   |                               |
                  NO                              YES
                   |                               |
                   v                               v
         Is principal opted-in?            Is principal opted-in?
                   |                                 |
           +-------+-------+                 +-------+-------+
           |               |                 |               |
          NO              YES               NO              YES
           |               |                 |               |
           v               v                 v               v
    Has ALL/Super    Has required        IAM PATH        Has required
    LF permission?   LF permissions?     (IsRegistered   LF permissions?
           |               |              =False)              |
       +---+---+       +---+---+         +---+---+        +---+---+
       |       |       |       |         |       |        |       |
      NO      YES     NO      YES       NO      YES      NO      YES
       |       |       |       |         |       |        |       |
       v       v       v       v         v       v        v       v
      ❌      ✅      ❌      ✅        ❌      ✅       ❌      ✅
     FAILS  SUCCESS FAILS  SUCCESS    FAILS  SUCCESS   FAILS  SUCCESS
            (ALL           (LF PATH)         (IAM)             (LF PATH)
            bypass)    

Why TRUNCATE fails without opt-in:

1. Principal is not opted-in
2. Lake Formation directs to IAM path
3. Glue requests temporary S3 credentials from Lake Formation
4. Lake Formation checks: Is principal opted-in? Answer: NO
5. Lake Formation does not provide credentials
6. Glue attempts to use role's IAM S3 permissions
7. Lake Formation intercepts the S3 request (location is registered)
8. Access denied - Lake Formation expects its own credentials

Why TRUNCATE works with opt-in:

1. Principal is opted-in
2. Lake Formation enforces Lake Formation permissions
3. Lake Formation verifies DELETE permission is granted
4. Glue requests temporary S3 credentials from Lake Formation
5. Lake Formation checks: Principal opted-in? Location registered? Has DELETE permission?
6. All conditions met - credential vending occurs
7. Lake Formation generates temporary credentials with s3:DeleteObject
8. Glue uses temporary credentials to delete S3 objects
9. TRUNCATE succeeds

Prerequisites

Before proceeding, verify your configuration:

1. Confirm your S3 location is registered with Lake Formation in hybrid access mode
2. Verify your Glue execution role has Lake Formation DELETE permission on the table
3. Verify your Glue execution role has IAM s3:DeleteObject permission in its IAM policy

Solution: Opt-in your Glue execution role

Choose one of the following methods to opt-in your Glue execution role for hybrid access mode.

Method 1: Using the console checkbox (recommended)

1. Open the Lake Formation console

2. In the navigation pane, choose Tables

3. Select your table

4. Choose Grant

5. For IAM users and roles, select your Glue execution role

6. For Table permissions, select DELETE, INSERT, and SELECT etc.

7. Select the checkbox Make Lake Formation permissions effective immediately

   Note: This checkbox opts-in the principal for hybrid access mode

8. Choose Grant

Method 2: Using the Hybrid access mode menu

1. Open the Lake Formation console
2. In the navigation pane, choose Hybrid access mode
3. Choose Opt-in principals
4. Choose Opt in principals
5. For IAM users and roles, select your Glue execution role
6. For Resources, select your database and tables
7. Choose Opt in

Method 3: Using the AWS CLI

Run the following command to opt-in your Glue execution role:

aws lakeformation create-lake-formation-opt-in \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::ACCOUNT_ID:role/GLUE_ROLE_NAME \
  --resource '{
    "Table": {
      "CatalogId": "ACCOUNT_ID",
      "DatabaseName": "DATABASE_NAME",
      "Name": "TABLE_NAME"
    }
  }' \
  --region REGION

Note: Replace ACCOUNT_ID, GLUE_ROLE_NAME, DATABASE_NAME, TABLE_NAME, and REGION with your actual values.

Verify the opt-in status

To confirm your Glue role is opted-in, run:

aws lakeformation list-lake-formation-opt-ins \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::ACCOUNT_ID:role/GLUE_ROLE_NAME \
  --region REGION

The output displays all resources for which the principal is opted-in.

Test the TRUNCATE command

Run your Glue job with the TRUNCATE command:

spark.sql("TRUNCATE TABLE database_name.table_name")

The command now succeeds because Lake Formation provides credential vending for opted-in principals.

Understanding credential vending

Credential vending in Lake Formation requires both conditions to be true:

1. S3 location is registered with Lake Formation
2. Principal is opted-in for hybrid access mode

When both conditions are met, Lake Formation generates temporary S3 credentials with the necessary permissions (s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket) based on the Lake Formation permissions granted to the principal.

Why IAMAllowedPrincipals works (not recommended)

If you previously granted ALL permissions to the IAMAllowedPrincipals group and TRUNCATE worked, this is because IAMAllowedPrincipals completely bypasses Lake Formation authorization. However, this approach:

• Bypasses Lake Formation governance
• Removes centralized access control
• Eliminates audit trails in Lake Formation
• Defeats the purpose of using Lake Formation

For proper governance, use the opt-in approach instead of IAMAllowedPrincipals.

Additional considerations

For cross-account scenarios: When opting-in all tables in a database for cross-account grants, you must also opt-in the database itself for the permissions to work correctly.

For DATA_LOCATION permission: Principals require DATA_LOCATION permission on the S3 location to create tables or databases pointing to that location, regardless of opt-in status.

For cleaning up IAM policies: After opting-in principals, you can optionally remove S3 permissions from their IAM policies since Lake Formation provides credentials through credential vending.

Related information

• Prerequisites
• Hybrid access mode - AWS Lake Formation
• How hybrid access mode works - AWS Lake Formation
• Setting up hybrid access mode - AWS Lake Formation
• Hybrid access mode considerations and limitations - AWS Lake Formation
• Integrating third-party services with Lake Formation - AWS Lake Formation