Require a partition filter in all Athena queries?

0

Does Athena have a way to enforce/validate that every query includes at least one WHERE clause that filters on a partitioned column? I'm looking for a way to prevent query authors from accidentally scanning their entire 2TB+ tables.

I've found the iceberg.query-partition-filter-required setting in Trino. Does Athena have anything similar?

Set to true to force a query to use a partition filter for schemas specified with iceberg.query-partition-filter-required-schemas. Equivalent catalog session property is query_partition_filter_required.

asked a month ago55 views
1 Answer
1
Accepted Answer

Athena does not have a provision to enforce a WHERE clause on partition columns in a query unfortunately. However, Athena workgroups do allow you to set data usage controls using which you can restrict the data scanned by queries.

There are two types of data usage controls that can be utilized:

  • The per-query limit: Allows you to specify the total amount of data scanned per query. Any query in the workgroup that exceeds this data limit gets canceled.
  • The workgroup-wide data usage alert: Allows you to specify thresholds on aggregate data scanned by all queries running in the workgroup within a specified time period (For example, hourly or daily aggregates). Please note that, with this limit, queries are not automatically canceled when a threshold is reached. But instead, it can be used to set alerts and push notifications to SNS topics depending on which you can take further actions (For example, invoking a Lambda function that disables the workgroup).

Please refer to the following document for more information regarding these data usage controls: https://docs.aws.amazon.com/athena/latest/ug/workgroups-setting-control-limits-cloudwatch.html

AWS
SUPPORT ENGINEER
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions