- Newest
- Most votes
- Most comments
There is a caveat yes. If you have a single table in us-east-1 which is STANDARD_IA and is intended to contain all TTL deleted items it will not work for this reason:
If TTL is invoked in us-west-2, the replicated write to us-east-1 will not contain the TTL identifiers { "userIdentity": { "type": ["Service"], "principalId": ["dynamodb.amazonaws.com"] }} and it will appear as a standard DeleteItem with no userIdentity. That means your STANDARD_IA table in a single region will not store all items which were evicted.
I think I might be getting confused now @Mani - haha! Let me go over what I understand about the configuration you're talking about.
Global Table with two regional replicas - us-east-1 and us-west-2. GT requires that both regional replicas have Streams enabled, and if you configure TTL for the GT, the TTL scanner will operate in each regional replica independently (there is no way to have TTL operating in one region but not the other). If an item is found by the scanner in us-east-1, the delete will be made locally and will appear in the event stream in us-east-1 with the TTL identity. That TTL delete will also be replicated to us-west-2, and in us-west-2 the event for that delete will appear in the stream without the TTL identity - it will look like any normal application-driven delete (originating in either region). The scanner in us-west-2 is also running, of course - it may also locate the same "expired" record and make the TTL delete in the local replica - and that will, again, be replicated to us-west-1 (and will look like an application-driven delete).
Because the regional replica TTL scanner is running in both region replicas, choosing to archive from TTL-identified deletes in just one of the two region replica streams is almost certainly going to give you an incomplete archive. I believe that your goal is to capture all TTL deletes from the GT in one (non-global) table, TabNV_IA (in us-east-1). To be sure that you capture all TTL expired items from the GT, I would suggest that you run a Streams processor in both us-east-1 and us-west-2. Each listens to the local replica's stream, and only looks for TTL identity deletes. That deleted item (view from before TTL delete) is written by the stream processor to TabNV_IA. The writes from the stream processor in us-west-2 will be cross-region, which should be fine. Also, some of the writes of these TTL deleted items could be duplicate (TTL scanners in each region replica locate the expired record at nearly the same time and both make the local TTL delete). You need to consider how you want to handle the duplicates - my guess is that you will be okay with just upserting (overwriting) in TabNV_IA if there's already an item present with that key.
Please let me know if I'm misunderstanding your requirements or your questions. Maybe Lee will have some additional thoughts or corrections on this.
Thanks Pete for such detailed explanation. However, still one of my point is being overlooked here. In this whole explanation, what is being talked about is "capturing TTL Deletes, be it in any region". I am talking about "having just one IA table (for archiving) in us-east-1 and capturing local deletes as well as TTL deletes of Replicated Writes (originating in us-west-2, replicated in us-east-1)." See, all replicated writes coming from other regions will be treated as local TTL deletes by TTL scanner in us-east-1 when they expire. Please correct me if I am wrong here. (Note: Due to Global Table replication, all records from other regions will pre-exist well before the TTL expiry happens) If I am correct, then having just 1 IA table in us-east-1 will capture all expired records without losing anything. Again, please let me know if there is a way to chat 1-o-1 or come over a call, so that I can explain better.
I see - yes, I was misunderstanding. I thought you wanted to capture TTL deletes exclusively. If you are fine with capturing all deletes, then I think just running the streams processor in us-east-1 should work pretty well as you suggest. There may be one nuance, which would be understanding what happens if an item is inserted and then immediately deleted (by the application) in us-west-2. GT replication is monotonic, but sparse. What I mean by this is, if an item rapidly goes from state A to state B and then state C (locally) in one region replica, the replicated state in another region may go direct from A to C (the B state may never exist in the replicated regions). GT tries to get to the point of matching state as quickly and directly as possible - there can be "skipping".
I think that if your application added an item in us-west-2 and then immediately deleted it locally, it's possible that the the item might never exist in us-east-1, and thus you'll never see any kind of delete for it in the us-east-1 stream. So be aware of that possibility - would actually be very interesting to test this! Adding it to my list of possible future blog topics :-)
PS - I am being cautious about setting up direct connection via chat or call as I'm actually an independent DynamoDB consultant. So, I need to be sure that I respect the guidelines of the re:Post community and wouldn't want to create an impression that my participation here is "spam". I hope you'll understand @Mani.
Relevant content
- Accepted Answerasked a year ago
- asked a year ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 7 months ago
Thanks for your reply Leeroy! But, in my setup, I do not have TTL enabled in us-west-2. So TTL will not be invoked at all. Only place TTL will get invoked is us-east-1. Items from us-west-2 will appear as replicated writes and hence land in us-east-1 IA table post expiration. Still, you see any caveats or a reason to have TTL triggers in each replica region ?
I think the part you seem to be missing is the TTL setting will be applied to all replicas, its not region specific for Global Tables V2019 which is the current version.
"TTL setting will be applied to all replicas, its not region specific" Does this mean that irrespective of we enable it on our table in specific region (say, us-west-2), records will get expired. In absence of a TTL trigger, TTL delete will be replicated without the mentioned TTL identifier and we will lose the record. But, TTL expiration will happen when record expiry is reached. While write will get replicated from us-west-2 region in us-east-1 region almost instantly (well before TTL expiration occurs). And I think, when these replicated writes get expired, they will be taken care by us-east-1 trigger (please confirm). Now, even if my us-west-2 records expire and not captured by TTL trigger (as one is not configured), then also I do not lose anything. Please correct me if my understanding is wrong.
If you are using Global Tables, then enabling TTL in one replica will enable it for all replicas. If you only archive data in a single region, then a TTL delete from the other region will be lost.
Alright, this is something which I guess not documented anywhere in AWS documentation that irrespective of TTL Enable property, enabling TTL for one replica, enables for all. Thanks for letting me know. However, I am still unable to understand the scenario in which a TTL delete will be lost. us-west-2 writes would get replicated almost instantly in us-east-1. While, TTL expiration will occur way later. How come TTL delete will get lost ? We already have all data replicated as part of us-east-1 table which is tied to IA table and TTL deletion of replicated records (from us-west-2 region) also get moved to IA table (as they are already part of us-east-1 table).
Please help me understand this better with an example pls.