dms replication task -target files overwrite

Question

hi
i have a DMS Replication task that copy data from DB to S3 folder , and save the data as csv. the task is with type : FULLLOAD and the TargetTablePrepMode is DROP_AND_CREATE

i want to schedule this task and import the complete table every time with overwriting the csv in the target.

if i understand correctly DMS is deleting the old file and creating new file instead. i have a risk that in the time between the file delettion and the new file creation , someone will query this table and will get an error or no data.

is there a way to prevent this case ? is there a detailed documentation about the duration no data will exist in the folder in such task ?

Thanks

subhashr · Answer

Implement a Staging Area: Instead of directly overwriting the production file, use a staging area in S3. Upload the new file to the staging area, and only after the upload is complete, swap it with the production file. This approach minimizes the window of potential data unavailability.

Organize your data using prefixes or folders in S3. For example, you could have a "current" folder and an "updating" folder. Write new data to the "updating" folder, and once complete, move it to replace the "current" folder.

dms replication task -target files overwrite

Relevant content