How to solve HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://bucket/raw/schema/table/file.parquet Rate exceeded (Service: AWSLakeFormation; Status Code: 400; Error Code: ThrottlingException...?

Question

I'm trying to execute queries in Athena that involve multiple tables using joins. However, when I attempted to run several queries simultaneously (8 queries), I encountered the error. To address this, I reduced the number of files in each table to a total of 356 files across all tables. Despite this, the issue persists.

What could be causing this problem, and how can I resolve it?

For context, I'm using the Boto3 client to call `athena_client.start_query_execution(...)` and then checking the status of each query every X seconds with `athena_client.get_query_execution(QueryExecutionId=_id)`.

Osvaldo Marte · Answer

Since running multiple queries simultaneously (8 in your case) leads to throttling, try reducing the number of concurrent queries. Start by running fewer queries at a time, such as 2 or 3, and gradually increase while monitoring for throttling exceptions.

💡 If you need to run multiple queries, batch them to reduce the number of concurrent queries. This can help avoid rate limits.

>💡 If you need to run multiple queries, batch them to reduce the number of concurrent queries. This can help avoid rate limits.

How to solve HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://bucket/raw/schema/table/file.parquet Rate exceeded (Service: AWSLakeFormation; Status Code: 400; Error Code: ThrottlingException...?

Contenido relevante