How to Deploy AI Models from S3 to ECS Using NVIDIA Triton Without Local Downloads?

Question

We have AI models that are pushed daily to an S3 bucket from an EC2 instance. We want to deploy these models in an ECS environment using NVIDIA Triton Inference Server, without downloading them locally to the ECS instances. Can you provide any documentation or information on how to set this up.

Joe Lin · Answer

One best practice pattern is to use Elastic File System (EFS) as the shared storage location.  It can be mounted by both EC2 instances and ECS managed containers/tasks running on either EC2/Fargate fleet type.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

Retain push to S3 bucket for version archival, and the daily published AI model can be stored in EFS file system instead.  It is recommended to create Availability Zone (AZ) specific mount targets for the file system to minimize cross-AZ data transfer latency and cost.

https://docs.aws.amazon.com/efs/latest/ug/accessing-fs.html

Existing EC2 can be used to retrieve older versions of the model from S3 persistent storage as needed.  Use the right-sized EFS volume as working data location.

How to Deploy AI Models from S3 to ECS Using NVIDIA Triton Without Local Downloads?

Add your answer

Relevant content