AWS Transfer Family: FTP To S3 Cron Job?

Question

Hi,

I am trying to transfer files from a remote FTP server into an S3 bucket.
It is expected that between 100-1000 new files will be uploaded to the FTP server daily, which would mean new files would have to be transferred to S3 each day.

The naming convention of the files includes a time stamp if that is relevant.

Note, I do not need to host an FTP server myself. Just need to transfer files.
Looking at docs it seems "AWS Transfer Family SFTP connectors" is a service for this use case.

I'm wondering if this is a good use case for AWS Transfer Family? 
Some sort of daily cron job to transfer only the newly added files in an FTP server to S3.

Steve_M · Accepted Answer

Yes, AWS Transfer Family sounds like a good fit for this use case. With one caveat - you mention FTP (without a leading or trailing S) in your question, and while AWS Transfer Family does support plain FTP this is only for internal-only connections - basically the source and the endpoint are in the same VPC, or peered VPCs, or over Direct Connect/VPN/etc.

https://repost.aws/knowledge-center/aws-sftp-endpoint-type

If the source is initiating a connection over the internet to a public endpoint (or even if it's within a VPC, but the endpoint has internet-facing access) then plain FTP cannot be used, only SFTP or FTPS.

If this will work for you, then in answer to your second question it should be easy to script this to run in cron. As far as the source is concerned it's just connecting to another SFTP (or FTPS or FTP) server, it doesn't know (and doesn't need to know) that there's an S3 bucket behind it.

Vinamra Jain · Answer

Hi! AWS Transfer Family provides fully managed [SFTP connectors](https://aws.amazon.com/about-aws/whats-new/2023/07/aws-transfer-family-sftp-connectors/) that can used to transfer files from remote SFTP servers into S3. We do not offer FTP connectors yet, and hence, you ill not be able to use Transfer Family for transferring files from remote FTP servers.

Also, to transfer files using SFTP connectors, you will need to provide the full file path (including the file name) when invoking the file transfer API. Currently, there isn't a way to directly specify "newly added files" for transfer, and you will have to provide the full file names for these new files when using SFTP connectors to transfer them from remote SFTP servers.

ShadowFox · Answer

can we get  files from sftp server hosted in private network outside aws and store it in S3, I beleive it is not possible today and this limitation is not mentioned anywhere in aws documenation.

AWS Transfer Family: FTP To S3 Cron Job?

Relevant content