Memory error in amazon sagemaker

0

Memory error occurs in amazon sagemaker when preprocessing 400 mb of data which is stored in s3. No problem in loading the data. Dimension of data is 8-9 million rows and 4 columns, but I get 7000 columns after applying One hot encoding. One tying to train de model I get the following memory error: MemoryError: Unable to allocate 946. MiB for an array with shape (123962752,) and data type float64

Notebook instance is ml.t2.medium. How to solve this issue?

jorge
asked 2 years ago244 views
1 Answer
1

Hello, without changing your code, one option would be to switch to an instance type having a larger amount of RAM memory. The usage pricing for each instance type is detailed on https://aws.amazon.com/sagemaker/pricing/

Now, there are other options to perform one-hot encoding and other Feature Engineering tranformations, like SageMaker Data Wrangler and AWS Glue DataBrew. References:

AWS
answered 2 years ago
profile pictureAWS
EXPERT
Chris_G
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions