An error occurred (403) when calling the HeadObject operation: Forbidden when downloading a file from S3 bucket

Question

Error is when executing s3_client.download_file(bucket, key, local_tar_file) in a  lambda function:

Here is the function in the lambda:

def download_incoming_file(bucket, key):
    logger.info(f'download_incoming_file... {bucket} - {key}')
    try:
        local_tar_file = '/tmp/' + os.path.basename(key)
        
        # Create S3 client with SSL enabled
        s3_client = boto3.client('s3',use_ssl=True)
        s3_client.download_file(bucket, key, local_tar_file)
        logger.info(f'downloaded incoming_file... {bucket} - {key} - {local_tar_file}')
    except ClientError as e:
        error_message = f'Error downloading files to S3 bucket : {bucket} file : {key}; Error : {str(e)}'
        logger.error(error_message)
        raise ExtractionError(error_message)
    return local_tar_file

here is the execution role policy looks like:

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::abc-rrr-xxxx-yyyy-incoming",
            ]
        },
        {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                 "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::abc-rrr-xxxx-yyyy-incoming/*"
            ]
        },
        {
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:kms:ap-east-1:123123141241:key/b0sfdasdf-4dfb-489a-8abd-8358761e0f9b"
            ]
        },
    ],
    "Version": "2012-10-17"
}

Here is what i have for S3 bucket policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "Deny Insecure Traffic",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::dchi-razor-prod-bishop-incoming/*",
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

I verified that bucket and key are correct, but not sure why I am getting this error:
Can someone please point out what is the issue here?

Answer

Hello.

What are the ACL settings for the target object?  
The bucket policy and IAM policy seemed fine, so is it possible that the object ACL is preventing access?  
https://repost.aws/knowledge-center/s3-bucket-owner-access  
https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl

I don't think it matters much, but since the version of the bucket policy you are using is old, I think it would be better to set it to "2012-10-17" as shown below.  
https://repost.aws/knowledge-center/s3-bucket-policy-for-config-rule  
```
{
  "Id": "ExamplePolicy",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSSLRequestsOnly",
      "Action": "s3:*",
      "Effect": "Deny",
      "Resource": [
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET",
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      },
      "Principal": "*"
    }
  ]
}
```

Answer

As well as Riku's suggestion.

What do you have defined for local_tar_file ?

With lambda you can use /tmp to store the downloaded file. Any where else within the function is likely to fail due to not having write access

Answer

As your bucket config and IAM is proper, I believe the issue is with "local_tar_file = '/tmp/' + os.path.basename(key)" where you are trying write into lambda's file system.

Answer

I am using VPC endpoint for S3 and after adding the policy statement for the S3 bucket in the s3 VPC endpoint policy, it started working.

An error occurred (403) when calling the HeadObject operation: Forbidden when downloading a file from S3 bucket

関連するコンテンツ