AWS Appflow dynamic filter

0

Hi Community, I am new to AWS Appflow. I am trying to extract SAP table data using AppFlow. I am able to create a connection. And then data was successfully extracted. However, this table has a watermark column using which we need to extract incremental records. Hence it is important that we set the filter dynamically i.e. on a daily basis it would fetch last 3 days data. Is it possible to make the filter dynamic? Or Should I call the AppFlow from a Glue job with passing the dates in the filter option as parameters?

bigbang
asked 9 months ago797 views
1 Answer
0

Hi Bigbang

AWS AppFlow has Incremental transfers, which seem to be what you are looking for: see https://docs.aws.amazon.com/appflow/latest/userguide/flow-triggers.html

When you select incremental transfer, Amazon AppFlow transfers only the records that 
have been added or changed since the last successful flow run. You can also select a 
source timestamp field to specify how Amazon AppFlow identifies new or changed 
records. For example, if you have a Created Date timestamp field, choose this to instruct 
Amazon AppFlow to transfer only newly-created records (and not changed records) since the
 last successful flow run. The first schedule-triggered flow will pull 30 days of past records 
at the time of the first flow run.

Same page details the Offset option

Optionally, you can add a time offset (t) to the time range for the incremental transfer. 
The flow run will import records that were created or changed between the previous flow 
run and the specified offset prior to the current flow run. This feature can be used to accommodate 
any latencies in the source systems in timestamping changes to records. By choosing a 
sufficiently large offset, you can avoid missing records that changed in the source application close 
to the run time of the scheduled flow.

If a schedule-triggered flow runs at time instances T0, T1, T2, and so on, then records that are new 
or have changed between T0 minus t and T1 minus t will be imported from the source at T1, and 
those that have changed between T1 minus t and T2 minus t will be imported from the source at T2.

So, by combining both, you should be able to achieve what you want.

Best, Didier

profile pictureAWS
EXPERT
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions