Amazon S3 Connector for PyTorch now supports writing checkpoints with PyTorch Lightning

1

The Amazon S3 Connector for PyTorch now supports saving PyTorch Lightning model checkpoints directly to Amazon S3, improving the cost and performance of your machine learning training jobs. PyTorch Lightning is an open source framework that provides a high-level interface for training with PyTorch. The Amazon S3 Connector for PyTorch automatically optimizes S3 requests to improve data loading and checkpoint performance for your training workloads. Saving PyTorch Lightning model checkpoints is up to 40% faster with the Amazon S3 Connector for PyTorch than writing to Amazon EC2 instance storage.

The Amazon S3 Connector for PyTorch delivers a new implementation of PyTorch Lightning's checkpoint primitive that you can use to save machine learning model checkpoints directly to Amazon S3. Model checkpointing typically requires pausing training jobs, so the time needed to save a checkpoint impacts overall training times. With this integration, you can save, load, and delete checkpoints directly from PyTorch Lightning training jobs to Amazon S3.

Amazon S3 Connector for PyTorch is an open source project. To get started, visit the GitHub page.

AWS
已提問 2 個月前檢視次數 547 次
沒有答案

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南