Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

0

I am using glue to move data from s3 and redshift into a data lake. I would like to use a combination of AWS glue and Athena to create a source to target mapping report. The goal is to make the report dynamic, so that if an element in the ETL process is changed (i.e. a table name is altered) the report will adjust accordingly to the respective changes that have been made. What is the best practice that has been used to complete this task? Or what solutions exist that help solve this problem.

Themen
AnalysenDatenbank
Tags
Amazon AthenaAnalysenAWS GlueDaten extrahieren, transformieren & ladenAmazon Redshift
Sprache
English
Andrew
gefragt vor 10 Monaten275 Aufrufe
1 Antwort
0

If it's just the table name, just a job argument and then use the argument in the code.
For something more sophisticated you probably need some parameters and code logic, or just build a SparkSQL query dynamically.

It's not common practice to combine Glue and Athena for the same task (e.g. mapping and saving the data), in this case it sounds like Glue is more suited for that ETL.

profile pictureAWS
EXPERTE
Gonzalo Herreros
beantwortet vor 10 Monaten
  • Andrew
    vor 10 Monaten

    Glue natively stores the source meta data in glue catalog. Is there a way to customize the crawlers to record target meta data in the same table? If so, what are the steps to do so?

  • Gonzalo Herreros EXPERTE
    vor 10 Monaten

    You could use a crawler and then add some of your own metadata, but if you run the table again it can be overriden

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt