How can we reconcile the number of records which AppFlow for SAP pulls using oData?

Question

How can we reconcile the number of records which AppFlow for SAP pulls from our SAP system? We would like to automate the reconciliation of number of records SAP is sending vs. the number of records ingested (landed in S3) for each data load using AppFlow?

NOTE: This is a common question asked by AWS customers. The AWS team is posting it to provide an answer that can benefit everyone.

**NOTE:** *This is a common question asked by AWS customers. The AWS team is posting it to provide an answer that can benefit everyone.*

Accepted Answer

It should be possible to automate “the reconciliation of number of records SAP is sending vs. the number of records ingested (landed in S3) for each data load”, this can be done through the means of lambda with the following methods :
 
To get the SAP OData record extraction:
1/ Non-ODP Based Extraction : use $count as per https://blogs.sap.com/2013/03/20/using-odatas-top-skip-and-count/
2/ ODP Based Extraction : use $inlinecount as per  https://help.sap.com/doc/saphelp_nw75/7.5.5/en-US/50/f4ee6253134d3cafa25b9444f0c5a9/frameset.htm
 
To get the Appflow execution records:
3/ https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/appflow.html#Appflow.Client.describe_flow_execution_records
 
Hope this helps.

How can we reconcile the number of records which AppFlow for SAP pulls using oData?

Contenido relevante