KMS Decrypt Call Frequency in Cross-Account S3 Queries with encrypted Objects

Question

Hi everyone, I am trying to set up cross account access for Glue catalog, I am using a customer-managed key in Glue and also set s3 encryption type to SSE-KMS and re-encrypt all current object.. my question is : If all S3 objects have to be re-encrypted with the new key, does it mean that every object needs to be decrypted when running a cross-account query? For example, lets say the query from the source account returns 100k objects from the destination account. Will the KMS Decrypt function be called once for accessing the Catalog in destination account, or will the function be called 100k times when accessing the results of the query?

Sandeep · Answer

Hello\
check these to resolve the issue:

1. Check IAM Permissions:

Verify that the role used in the source account has the necessary permissions to access the Glue catalog and S3 objects in the destination account. Use IAM policy simulator to validate permissions.

2. Verify KMS Key Access:

Ensure the role in the source account has proper access (Decrypt) to the customer-managed KMS key used for S3 encryption in the destination account.

3. Monitor KMS Decrypt Calls:

Use AWS CloudTrail to monitor KMS Decrypt calls during your cross-account Glue queries. This helps identify if Decrypt is being called excessively.

4. Analyze Query Efficiency:

Review your Glue query and see if it retrieves unnecessary data. Consider filtering or projecting specific columns to reduce the number of objects accessed.

mahankali vamseekrishna · Answer

Hello,

IF you suspect issues with KMS Decrypt calls or Glue catalog access, here are some troubleshooting steps:
**CloudTrail Logs**: Enable CloudTrail logging for the KMS key and Glue service in both accounts. This will provide detailed information on KMS API calls (including Decrypt) and Glue catalog interactions.

**VPC Flow Logs: **If applicable, consider enabling VPC Flow Logs to monitor network traffic between your Glue ETL environment and the S3 buckets. This can help identify potential network bottlenecks.

**Amazon CloudWatch Metrics**: Monitor relevant CloudWatch metrics, such as KMSDecryptionErrors and GlueCatalogGetTableErrors, to identify any errors or performance issues.

**Glue Job Logs**: Review Glue job logs for any warnings or errors that might indicate decryption problems.

Leo K · Answer

Yes, when the objects in S3 are encrypted with SSE-KMS, each object will have to be decrypted whenever its contents are accessed.

If you've enabled the bucket key option for SSE-KMS and you access multiple objects encrypted with the same bucket key within a short period of time, S3 will attempt to have KMS decrypt the bucket key once and reuse it to decrypt multiple object keys within the brief period of time that the decrypted bucket key is retained by S3, without calling KMS repeatedly. The exact behaviour will depend on timing and other variables, so whether the number of calls to KMS for a given query will be one or three, for example, won't be completely predictable.

KMS Decrypt Call Frequency in Cross-Account S3 Queries with encrypted Objects

Relevant content