Memory requirements

Hi Jason,
Thanks for reaching out. Using 16GB of RAM will work for transferring less than 10 million files per task. However, you may run into other limits or lower performance. I've also noted this as a feature request for lower memory requirements. Please let us know if you have additional feedback, and feel free to send me a private message if you have more questions about your specific account/setup.
Regards,
Olga Kogan
AWS DataSync

Hi.
Thanks for the response.
We have about 25 million files total, about 9TB.
Each task would copy over around 6 million files, 1.8TB.

Each task would copy less than 10 million, after the second task there would be more than 10 million files in the S3 bucket.
With this scenario, do you think 16GB would still be workable?

We are doing a full systems migration to AWS, as such we can not purchase any new hardware as much of our existing infrastructure will be freed up once the migration is done.
But without the data being migrated, the servers can't be made available.
Bit of a chicken and the egg problem!

One other question, for the logging, is there any way to capture the list of files that were copied?

Cheers,
Jason

Hi Jason,
As I mentioned before, using 16GB of RAM will work for transferring the number of files per task you described. However, as this is not our recommended configuration, you may run into other limits or lower performance. If you run into issues, please contact support and we'll take a look. We don't currently generate a list of all transferred files. I've noted your suggestion and we'll consider it as we plan new features.
Regards,
Olga Kogan
AWS DataSync

Hi Olga,
Thanks again for your reply.

We are going to give it a try and see how it goes.

I have put in a request for more memory, maybe I'll get lucky!

Thanks for noting my requests for features, both would be welcome additions.

I know it is early in the product life cycle.

Cheers,
Jason

Hi Jason,

We have just released the ability to log all transferred files:
https://aws.amazon.com/about-aws/whats-new/2020/04/aws-datasync-enhances-monitoring-capabilities-with-file-level-logging/

Best,
Olga