AmazonS3Exception error when running CTAS using Athena engine version 3

0

We are experiencing the following issue, that is blocking us from upgrading to Athena engine version 3. It's important to note that the query runs successfully when using Athena engine version 2. The S3 bucket being used denies uploads of unencrypted objects, in case that could be relevant.

Example query:

CREATE TABLE ctas_1772133c_00c9_440e_934e_c35ac928fdcd WITH (
    format = 'JSON',
    external_location = 's3://athena-query-results-123412341234/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd/'
) AS
SELECT name
FROM users
LIMIT 10

Error message received (masked, except for the request/query ids):

Error committing manifest file com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: NAPVPKQ1F1BHQHF8; S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=; Proxy: null), S3 Extended Request ID: jeovQuVvA/yuDyjNXg+K10z9oJDxxscRsdYO6A+rd53AkT/tq+ZlxDYwfMazypczaCKmuO8rebo=. You may need to manually clean the data at location 's3://athena-query-results-123412341234/Unsaved/2022/11/28/tables/ctas_1772133c_00c9_440e_934e_c35ac928fdcd' before retrying. Athena will not delete data in your account. This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 13a43d53-33d6-4536-98c8-ad116bc12637

This is the S3 bucket policy, for the bucket where data should be saved to:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::athena-query-results-123412341234/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}
  • Are you trying to overwrite the table into the old S3 location? Given this is an EXTERNAL table, the CREATE TABLE AS should be in a different S3 location compared the source table location.

  • This is always a different and unique ctas_<random_uuid> value for both the table name and the S3 external location, so it never collisions with an existing S3 path.

  • We are also facing same issue. I could not understand why such a core thing not fixed even after these many days. No one wants to write data without encryption.

posta un anno fa408 visualizzazioni
Nessuna risposta

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande