I do a crawler to load all my S3 csv files to Glue Data Catalog. Now I want to create a glue job to execute ETL (create and drop temporary tables, select and insert data to tables in Data Catalog) But in the glue job as a python shell I have to split my sql statements to execute one by one. With the following code, I got an error.
client = boto3.client('athena')
client.start_query_execution(QueryString = """drop table if exists database1.temptable1 ;CREATE EXTERNAL TABLE IF NOT EXISTS temptable1(id int ) """, ResultConfiguration = config)
Is there any way to run multiple sql statements in glue job?
AWS OFICIALAtualizada há 2 anos
To be especific, what you want is to run multiple queries on Athena using boto3, doing it from a Glue shell is circumstantial (you could do it from your computer and it would be the same). I don't think it's possible. Maybe try with with pyathena instead