- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
SageMaker Fast file mode streams the data directly from S3 when you access the file. From an usability perspective you will still access the files as if they are on disc and SageMaker makes sure to stream the file from S3 when accessed. For your use case using File Mode which does the full copy rather than streaming will be better approach as the initial copy is much faster for datasets less than 100 GB. Please refer to the below blog to determine the right option for your training
In the short term, I can deal with the default File mode. However, in the long term, I may need the Fast File mode (I didn't reach 100 GB of data yet). I was expecting to be working with a small example of nearly 30 GB, that's why I do not understand why it's not working, especially when I can switch from File to FastFile without changing the code.
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor einem Jahr
- AWS OFFICIALAktualisiert vor 2 Jahren
Do you have a large number of files ?
Ye, in those 27 GB, I have 266 folders with 4 numpy files each. Each folder is about 100 MB of data.