List, Delete operations with SFTP Connectors

0

I'm setting up an SFTP connector to sweep files from a client's server. The workflow is as follows, get the list of files available to download, trigger the download using the SFTP connector, wait for the connector to download the file, trigger the delete operation on the remote server. To make this work, I'm using Lambda functions for each of the operations and having to make individual SFTP connections for each file download and delete. In my use case, I have close to 200 files in the server and making 200 connections to download and 200 connections to delete the files seems like a lot.

Is there a way to create a workflow to trigger the connector (using EventBridge or Lambda trigger) with the external SFTP sweep path and delete permissions flag and for the service to manage sweeping subdirectories, download all available files and then delete them if the permission is enabled? This would be great if it is offered as a managed service in AWS. One other thing to manage would be to handle files in transit to the remote server, handling currently in Lambda.

2 Answers
1
Accepted Answer

Hi! You can design a workflow such that you use SFTP connectors to List files (StartDirectoryListing operation). Once file list is generated, connector emits an event to EventBridge, and you can use this event to trigger a lambda function that parses the file list and request each file path for transfer using the SFTP connectors' StartFileTransfer operation. Connectors also emit an event to EventBridge after eachfile is transferred, so that event can be used to trigger a custom lambda to delete files from remote server. Currently, SFTP connectors provide operations to List and to Transfer files, but not to delete files.

AWS
answered 8 days ago
  • Exactly, the delete files functionality is key here and it is unavailable. Will wait until that is enabled on the connector service.

-1

Consolidate your file operations into batches using a single SFTP connection for downloading and deleting multiple files at once, reducing the overhead of establishing numerous connections. This can significantly improve efficiency and reduce the overall processing time.

profile picture
EXPERT
answered a month ago
  • SFTP connectors (service offered by AWS) doesn't allow batch file operations or delete capabilities. I've reviewed the documentation available.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions