Glue - Change Schema - Dropdown for target key or custom visual transform

0

In our ETL process we are building out a pipeline where someones job is to take input files (ex. csv) and map the columns to existing column names. After the mapping is complete a glue workflow will perform all of the needed data post processing and eventually loading the data into our database. We will be ingesting 3k+ data sources.

The "Change Schema" widget almost works as intended. The only hangup is that the "Target key" value is a manual input. Ideally this would be a populated list containing column names from our database but can alternatively be a static list that we update as the schema changes.

So there's two questions:

  1. Is it possible to specify a dropdown list for "Target key" using the Change Schema visual transform?
  2. If not, is it possible to create a multi column custom visual transform? All I need in the transform is two columns side by side. One for "Source key" and one for "Target key" where target key is a dropdown list. The issue I am running into here is that I can't seem to create a multi-column json template. We don't even need to select the data type at this stage.
asked 25 days ago177 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions