Using Glue for dimensional model ETL into Redshift


A customer is wondering if they can use Glue for their dimensional model ETL. Would it be able to populate the dimensions and facts and load it into Redshift, or would they need to create a staging table in Redshift and then populate their dimensions and facts via querying with surrogate keys?

I don't see why Glue wouldn't work for a dimensional model schema, but I'm having a really hard time finding sources and information about it.

asked 4 years ago812 views
1 Answer
Accepted Answer

Glue can definitely be used for loading dimensional data into Redshift. Approach will depend on what kind of dimension it is (SCD Type). And you can certainly generate surrogate ids in Glue. Example: I have used this in the past.

def customer_id(custid):
    x = int(str(hashlib.md5(custid.encode()).hexdigest()[:10]),16)
    x = int(x)
    return x

However, make sure you follow the logic consistently across different datasets to produce consistent surrogate ids.

Ideally, a staging table should be present and from staging to main table, you can govern the logic through redshift procedure or plain SQL depending upon the complexity.

answered 4 years ago
profile picture
reviewed 8 days ago
profile picture
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions