How to partition files in AppFlow S3 destination?

0

Hi, I have successfully configured an AppFlow flow to import data from Sharepoint to an S3 bucket, using AWS console. Data in the bucket now appears structured as follows:

  • MyFlowName/
    • ExecutionId/
      • filename01.txt
      • filename02.txt
      • ...
    • ExecutionId (Json file containing mapped data)

I need to change this structure, because I need the files filename01.txt, filename02.txt ... stored in a known path inside the bucket (I do not want having randomly generated ExecutionId in the path). In the docs it seems possible but I do not find the "Partitioning" section in the AppFlow console. Is the documentation out of date? I also tried to modify my flow configuration using CLI but, following the cli-docs and using the update-flow API: aws appflow update-flow --flow-name FirmwareFromSharepointToAws- --profile emak --destination-flow-config-list '[{"connectorType": "S3","destinationConnectorProperties": {"S3": {"bucketName": "emak-firmware-omnys","s3OutputFormatConfig": {"fileType": "JSON","prefixConfig": {"pathPrefixHierarchy" : ["SCHEMA_VERSION"] ,"prefixType" : "FILENAME"},"aggregationConfig": {"aggregationType": "SingleFile"}}}}}]' --source-flow-config '{"connectorType": "CustomConnector","apiVersion": "v1.0","connectorProfileName": "Sharepoint_D365foPRD_Firmware"

I got an error says that the parameter "pathPrefixHierarchy" is not valid (even if it is specified in the doc and seems to be the one to use to avoid having executionId in the path). Is the CLI documentation out of date as well?

Seems that with the current AppFlow release is not possible to change how data are stored in S3. Please, can anybody tell me if is possible do this with AppFlow or if I need to use another solution? Thanks in advance

Gab
asked 4 months ago66 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions