By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

execute multiple sql statements against data catalog tables

0

I do a crawler to load all my S3 csv files to Glue Data Catalog. Now I want to create a glue job to execute ETL (create and drop temporary tables, select and insert data to tables in Data Catalog) But in the glue job as a python shell I have to split my sql statements to execute one by one. With the following code, I got an error.

client = boto3.client('athena') client.start_query_execution(QueryString = """drop table if exists database1.temptable1 ;CREATE EXTERNAL TABLE IF NOT EXISTS temptable1(id int ) """, ResultConfiguration = config)

Is there any way to run multiple sql statements in glue job?

  • Gonzalo Herreros EXPERT
    a year ago

    To be especific, what you want is to run multiple queries on Athena using boto3, doing it from a Glue shell is circumstantial (you could do it from your computer and it would be the same). I don't think it's possible. Maybe try with with pyathena instead

Topics
AnalyticsDatabase
Tags
Amazon AthenaAWS GlueExtract Transform & Load Data
Language
English
rePost-User-3320844
asked a year ago55 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content