Backfillig of timestream database

Question

I'm trying to backfill old data into timestream. Have tried different interfaces (.NET-SDK and Console/Batchload). This ends up always with :"The record timestamp is outside the time range of the data ingestion window.".

,"errors":[{"reason":"The record timestamp is outside the time range of the data ingestion window.","lineRanges":[[2,35959],[36989,40997],[43010,47047],[49082,53096],[55168,59177],[61243,71257],[73331,77336],[79350,83355],[85361,89367],[91374,95365],[97372,101377],[103383,115410],[117430,121379],[123386,133410],[135417,143433],[145440,149449],[151349,301520]]},

My data are max. 3 years old. I've activated (in the console): magnetic store write , magnetic storage duration is set to 5 years and memory storage to 1 hour. This backfill-functionality is a basic need for migrations into/from timestream.

Another finding: https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html

It says " You can then ingest data into the table using the same mechanism used for writing data into the memory store. Amazon Timestream for Live Analytics will automatically write the data into the magnetic store based on its timestamp."
But the data are still rejected by timestream.

ERROR: Rejected Record reson: The record timestamp is outside the time range [2024-04-11T08:46:40.278Z, 2024-08-24T14:03:30.681Z) of the data ingestion window., Index 0.

This date  (2024-04-11T08:46:40.278Z,) is the creation date of the table.

What is the aws recomended way to deal in that case ? Is Timestream only useable for freshly generated data ?

Sethu Srinivasan · Answer

If you are still facing this issue, can you pls report this issue via AWS support? 
https://aws.amazon.com/contact-us/

Thanks!

Basel Mohamed · Answer

try this code instead, and replace it with your Vars and values:

```
using Amazon.TimestreamWrite;
using Amazon.TimestreamWrite.Model;

public async Task UpdateTimeStreamTableForHistoricalData(string databaseName, string tableName)
{
    var client = new AmazonTimestreamWriteClient();

var request = new UpdateTableRequest
    {
        DatabaseName = databaseName,
        TableName = tableName,
        MagneticStoreWriteProperties = new MagneticStoreWriteProperties
        {
            EnableMagneticStoreWrites = true,
            MagneticStoreRejectedDataLocation = new MagneticStoreRejectedDataLocation
            {
                S3Configuration = new S3Configuration
                {
                    BucketName = "your-s3-bucket-for-rejected-data",
                    ObjectKeyPrefix = "timestream-rejected-data/"
                }
            }
        }
    };

try
    {
        var response = await client.UpdateTableAsync(request);
        Console.WriteLine($"Table updated successfully. Status: {response.Table.TableStatus}");
    }
    catch (Exception e)
    {
        Console.WriteLine($"Error updating table: {e.Message}");
    }
}
```

Backfillig of timestream database

Relevant content