how to save a file while running an etl job in a fargate task?

0

I have a python code for an etl job , i am planning to run as a fargate job. I am saving a docker container in ecr and pulling that to run my task, but i need to temporarily save a text file while running that container. i know in lambda, i think we can save a file to temporary folder as such /tmp/somefile.txt, i assume as lambda or a fargate task will run in some ec2, it should be the same?

asked 2 years ago1754 views
2 Answers
0

Yes, you are right. You can store the files temporarily in /tmp folder like you do in any EC2 instance, but it cannot exceed 20GiB with default settings. Please read this doc here.

Also, if you want to persist the files, you could use EFS mounting.

profile pictureAWS
SUPPORT ENGINEER
answered 2 years ago
0

To add to the answer, you can definitely indeed store files locally on the filesystem in similar way to lambda / EC2. Your image + local content can go up to 20GB "for free" with every task. You can go up to 200GB of that NVMe goodness, but you have to pay for the storage above 20GB (also you have to define that at the task definition level, it doesn't magically add storage for you).

But instead of EFS I'd much more recommend something a little more modern and use S3, as if you needed to do any form of automation (i.e. trigger a lambda when that temporary file is created/updated) then that's easy to do, whereas EFS won't give you that. But that very much depends on your IO pattern.

EFS requires also a little more involvement in the infrastructure. Not too much but substantially more than S3.

profile picture
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions