I currently have a DynamoDB global table set up with encryption at rest configured using an AWS managed key (not owned by Amazon, so KMS charges apply).
My service that hits the DynamoDB table is only running in us-east-1, but the global table replicates data to us-west-2 as part of the disaster recovery strategy.
Looking at recent months in AWS Costs Explorer, I noticed that there are significantly more KMS requests coming from us-west-2 than us-east-1:
US East (N. Virginia)
AWS Key Management Service us-east-1-KMS-Requests
$0.03 per 10000 KMS requests in US East (N. Virginia) 146,390,252.000 Requests
US West (Oregon)
AWS Key Management Service us-west-2-KMS-Requests
$0.03 per 10000 KMS requests in US West (Oregon) 272,575,473.000 Requests
Looking at CloudTrail, it looks like a majority of these calls are Decrypt operations related to the DynamoDB replication, similar to:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"arn": "arn:aws:sts::[account]:assumed-role/AWSServiceRoleForDynamoDBReplication/[id]",
"invokedBy": "replication.dynamodb.amazonaws.com"
},
"eventTime": "2022-11-16T04:26:41Z",
"eventSource": "kms.amazonaws.com",
"eventName": "Decrypt",
"awsRegion": "us-west-2",
"sourceIPAddress": "replication.dynamodb.amazonaws.com",
"userAgent": "replication.dynamodb.amazonaws.com",
"requestParameters": {
"encryptionContext": {
"aws:dynamodb:tableName": "[table-name]"
},
"encryptionAlgorithm": "SYMMETRIC_DEFAULT"
},
"responseElements": null,
"readOnly": true,
"resources": [
{
"type": "AWS::KMS::Key",
"ARN": "arn:aws:kms:us-west-2:[account]:key/[key-id]"
}
],
"eventType": "AwsApiCall",
"managementEvent": true,
"eventCategory": "Management"
}
Since us-west-2 is really just a replication destination with no active services hitting it, it's curious to me that there are so many calls to Decrypt using the us-west-2 key as the resource, unless there is something in the replication process that encrypts it using the us-west-2 key prior to replication, it gets sent over, and then needs to be decrypted to finish the replication process and add it to the table?
After coming across multi-region KMS replica keys, it seems like they could be a good candidate to address these large number of KMS requests, though not knowing where these Decrypt calls fall in the process makes it difficult to say for sure.
Given this setup, I have the following questions:
- Does a large number of KMS Decrypt API calls make sense for a backup region that only gets replications and doesn't really provide data to a service?
- Does a multi-region KMS replica key make sense to help address these KMS API call counts?
- Is there danger/issues with swapping the AWS managed KMS key to a multi-region KMS replica key on a large table (6,000,000,000 records, ~3TB in size)?
Probably worth reaching out to your TAM for deeper dive and getting authoritative answer. But could you check if the KMS key used for your replicated table in us-west-2 is actually AWS-managed key? You should be able to reference the key-id by going into KMS console in us-west-2 and look under "AWS managed keys" and find aws/dynamodb. Also, you should be able to swap key seamlessly without downtime per: https://docs.aws.amazon.com/kms/latest/developerguide/services-dynamodb.html#dynamodb-encrypt
Unfortunately not all accounts have Enterprise Support plans with TAM's who they can contact. @jkh can you let me know which version of Global Tables you are using, v2017 or v2019? https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/globaltables.DetermineVersion.html
@kumo-hiyori Yes, the KMS key used for the replicated table matches the key in KMS for aws/dynamodb in the us-west-2 region.
@Leeroy-Hannigan The Global Tables version is 2019.11.21.