Appflow INCREMENTAL extraction from SAP - How to keep CDC order in the incremental extractions?

0

HI all,

We have implemented few AppFlow feeds to extract data from SAP OData endpoint and ingest them on S3. Since most of the tables contain millions of rows, we are using INCREMENTAL extractions to transfer just the modified rows.

In order to re-construct on AWS side the tables, we need to apply deltas on AWS keeping the order of the changes (delete, update, create, etc.) , but to do so, we need the changes sorted by their execution date. Appflow is creating different files for the delta, although we follow the order in which the files are created by Appflow on S3 to apply the deltas (the order is taken from the date and time of file generation present in the name), there are cases where the order is not respected (e.g. delete of a record in file written before the file containing the update of the record).

Do you have any idea of how to keep the order of the records extracted by INCREMENTAL CDC?

Thanks, Luca

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions