How to solve HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://bucket/raw/schema/table/file.parquet Rate exceeded (Service: AWSLakeFormation; Status Code: 400; Error Code: ThrottlingException...?

I'm trying to execute queries in Athena that involve multiple tables using joins. However, when I attempted to run several queries simultaneously (8 queries), I encountered the error. To address this, I reduced the number of files in each table to a total of 356 files across all tables. Despite this, the issue persists.

What could be causing this problem, and how can I resolve it?

For context, I'm using the Boto3 client to call athena_client.start_query_execution(...) and then checking the status of each query every X seconds with athena_client.get_query_execution(QueryExecutionId=_id).

トピック

分析

タグ

Amazon Athena AWS Lake Formation

言語

English

David

質問済み 7ヶ月前334ビュー

1回答

新しい順
投票が多い順
コメントが多い順

これらの回答は役に立ちましたか？コミュニティがあなたの知識を活用できるように、正解に賛成票を投じてください。

Since running multiple queries simultaneously (8 in your case) leads to throttling, try reducing the number of concurrent queries. Start by running fewer queries at a time, such as 2 or 3, and gradually increase while monitoring for throttling exceptions.

💡 If you need to run multiple queries, batch them to reduce the number of concurrent queries. This can help avoid rate limits.

エキスパート

Osvaldo Marte

回答済み 7ヶ月前

David
7ヶ月前
Okay, thanks for your answer, Osvaldo. Do you know which limit can I check in AWS that is raising those throttling exceptions?
Osvaldo Marte エキスパート
7ヶ月前
For Amazon Athena, you can check the service quotas under Per Account API Call Quotas. The default limits are:

StartQueryExecution and StopQueryExecution: 20 calls per second

GetQueryExecution and GetQueryResults: 100 calls per second

ℹ️ If you use any of these APIs and exceed the default quota for the number of calls per second, or the burst capacity in your account, the Athena API issues an error similar to the following: "ClientError: An error occurred (ThrottlingException) when calling the <API_name> operation: Rate exceeded. Reduce the number of calls per second, or the burst capacity for the API for this account.

How to solve HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://bucket/raw/schema/table/file.parquet Rate exceeded (Service: AWSLakeFormation; Status Code: 400; Error Code: ThrottlingException...?

関連するコンテンツ