By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Use EvaluateDataQuality with PySpark DataFrame instead of Glue DynamicFrame?

0

Is there a way to use the class https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-EvaluateDataQuality.html with a PySpark DataFrame instead of a Glue DynamicFrame (and also without conversion to it)? I noticed, that working with Spark only is much more stable and significantly faster than working with Glue DynamicFrames. Therefore I would like to omit Glue DynamicFrames completely in my code.

Topics
Analytics
Tags
AWS Glue
Language
English
Thomas Mueller
asked 6 months ago153 views
1 Answer
0

No, added value features up to now have never been added to the standard DataFrame.
What you can do is convert to DynamicFrame just to evaluate the data quality and leave the rest of the code the same (or convert back to DataFrame), the overhead of converting from an back on most cases is minimal.

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content