Is there a difference in writing new record compared to overwriting a record in Timestream when we are writing to magnetic store?

Overwriting existing records (creating new versions for records with the same dimensions and timestamp) in Timestream's magnetic store will have minimal impact on the 6-hour write-to-read latency compared to writing entirely new records.

Here's why:

Eventual Consistency: Both new writes and overwrites go through the same eventual consistency process, leading to a similar 6-hour latency for becoming readable.

Focus on Partition Management: The 6-hour latency is primarily due to managing partitions within the magnetic store. Timestream needs to ensure data consistency and optimize storage by potentially creating new partitions or merging existing ones when writes occur. This applies to both new and overwritten records.

Impact on Performance:

However, there might be a slight difference in performance for the write operation itself:

New Writes: Require creating new entries within a partition. This might involve some additional overhead compared to overwrites. Overwrites: Modify existing entries within a partition. This could potentially be faster as it avoids creating new entries.

Overall:

The difference in write operation speed itself is likely negligible compared to the 6-hour eventual consistency latency. Both new writes and overwrites will experience similar delays in becoming readable.

Focus on Use Case:

When deciding between new writes and overwrites, focus on your specific use case: 1/ New Data Points: If you're adding entirely new data points with unique timestamps, new writes are the way to go. 2/ Updating Existing Data: If you need to update existing data points with the same timestamp, overwrites are more efficient as they avoid creating duplicate entries. Remember, the eventual consistency latency of 6 hours remains the primary factor for both scenarios.