same glue job running differently when used sample method

1

I have a csv file with 5 million rows in my s3. I used crawler on the file. I have a custom transformations in my glue job. My issue is, if I use create_dynamic_frame_from_catalog(), it is running very slow, where as if I use create_sample_dynamic_frame_from_catalog() with max sample limit as 5 million, it is working much faster. Why it is happening. i want to speed up the job without using the sample method.

1개 답변
1

Hello, based on the documentation, what parameter did you configure for create_sample_dynamic_frame_from_catalog function ? num is the one that defines the maximum number of records to be fetched.

Overall, regarding the job performance, there are a couple of strategies, like changing WorkerType and NumberOfWorkers parameter on the Job.

This blog post is also handy: https://aws.amazon.com/blogs/big-data/best-practices-to-scale-apache-spark-jobs-and-partition-data-with-aws-glue/

AWS
답변함 2년 전
profile pictureAWS
전문가
Chris_G
검토됨 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠