En utilisant AWS re:Post, vous acceptez les AWS re:Post Conditions d’utilisation
AWS re:Postre:Post

Use EvaluateDataQuality with PySpark DataFrame instead of Glue DynamicFrame?

0

Is there a way to use the class https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-EvaluateDataQuality.html with a PySpark DataFrame instead of a Glue DynamicFrame (and also without conversion to it)? I noticed, that working with Spark only is much more stable and significantly faster than working with Glue DynamicFrames. Therefore I would like to omit Glue DynamicFrames completely in my code.

Sujets
Analytique
Balises
AWS Glue
Langue
English
Thomas Mueller
demandé il y a 6 mois164 vues
1 réponse
0

No, added value features up to now have never been added to the standard DataFrame.
What you can do is convert to DynamicFrame just to evaluate the data quality and leave the rest of the code the same (or convert back to DataFrame), the overhead of converting from an back on most cases is minimal.

profile pictureAWS
EXPERT
Gonzalo Herreros
répondu il y a 6 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions

Contenus pertinents