Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

I am using glue to move data from s3 and redshift into a data lake. I would like to use a combination of AWS glue and Athena to create a source to target mapping report. The goal is to make the report dynamic, so that if an element in the ETL process is changed (i.e. a table name is altered) the report will adjust accordingly to the respective changes that have been made. What is the best practice that has been used to complete this task? Or what solutions exist that help solve this problem.

주제

분석 데이터베이스

태그

아마존 Athena 분석 AWS Glue 데이터 추출, 변환 및 로드 아마존 Redshift

언어

English

Andrew

질문됨 10달 전275회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

이 답변이 도움이 되었나요?커뮤니티가 여러분의 지식을 활용할 수 있도록 정답을 찬성하세요.

If it's just the table name, just a job argument and then use the argument in the code.
For something more sophisticated you probably need some parameters and code logic, or just build a SparkSQL query dynamically.

It's not common practice to combine Glue and Athena for the same task (e.g. mapping and saving the data), in this case it sounds like Glue is more suited for that ETL.

전문가

Gonzalo Herreros

답변함 10달 전

Andrew
10달 전
Glue natively stores the source meta data in glue catalog. Is there a way to customize the crawlers to record target meta data in the same table? If so, what are the steps to do so?
Gonzalo Herreros 전문가
10달 전
You could use a crawler and then add some of your own metadata, but if you run the table again it can be overriden

Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

관련 콘텐츠