AWS Builder Center: Learn, Build and Connect with builders in the AWS community
AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
A comparison of AWS Glue Data Catalog views and AWS Glue Iceberg materialized views
Compare Glue Data Catalog views with Glue Iceberg materialized views, and learn when to use each for governed data sharing, query performance, freshness, and cost.
Overview
AWS Glue Data Catalog views and Glue Iceberg materialized views both provide a reusable SQL interface to data, but they solve different problems:
- A Data Catalog view is a virtual, multi-dialect view. It stores SQL definitions and evaluates them when a consumer runs a query.
- An Iceberg materialized view is a managed table. It stores precomputed query results in Apache Iceberg format and can update those results on a configured schedule or through a manual refresh.
Choose a Data Catalog view when your primary requirement is a governed abstraction over source data. Choose an Iceberg materialized view when your primary requirement is to reduce the latency and repeated compute cost of frequently executed analytical queries.
Comparison at a glance
| Consideration | AWS Glue Data Catalog view | Apache Iceberg materialized view |
|---|---|---|
| Permissions model | • Works only with Lake Formation permissions model • Lake Formation Tags are supported • Lake Formation data cell filters are not supported • Definer role must have SELECT on all source tables and Create Table permission on the database where view is created | • Works with both Lake Formation and IAM only permissions • Lake Formation Tags are supported • Lake Formation data cell filters are not supported • Role creating the MV should have full table access on the source tables and access to write on the target MV location |
| Engine level support | • Athena, Redshift Spectrum and Spark dialects available with same account definer role • Athena and Spark dialects available with cross account definer role • You can create in any engine, add dialects from other engines and query in all supported engines • Spark clusters are required in FGAC configuration for Reads | • Can create only in Spark engine (Athena Spark, EMR, Glue Spark) • Can query in Athena SQL • Spark clusters are required in FTA configuration for Reads |
| Highlight | • Target – Available only for integrated AWS analytics services as query engines • Source - Supports any Glue table format based in S3 • Not supported for federated catalog tables, including S3tables | • Target - Iceberg format • Supports any Iceberg compatible query engine • Source - S3 based Iceberg tables and S3tables |
| Data storage | Stores metadata and SQL definitions; does not store query results | Stores precomputed results as an Iceberg table in Amazon S3 or Amazon S3 Tables |
| Query execution | Runs the view SQL against source tables at query time | Reads previously computed results |
| Freshness | Reflects source data when the query runs | Eventually consistent; depends on scheduled or manual refresh completion |
| Main purpose | Governed sharing, SQL abstraction, and a common catalog object across engines | Query acceleration and managed transformation pipelines |
| Source formats | Standard Data Catalog tables in formats supported by the query engine – Hive tables, Hudi, Delta Lake, and Iceberg tables in Lake Formation-registered S3 locations. | Source tables must be Apache Iceberg tables registered in the Data Catalog |
| Consumer permissions | Consumers can receive SELECT on the view without access to source tables | Consumers can receive SELECT on the materialized view without access to source tables |
| Cost profile | No separate result storage or refresh compute, but each query processes source data | Adds result storage and managed refresh compute, but can reduce repeated-query compute |
| Best fit | Governed logical interfaces and current results | Repeated, expensive queries with a defined freshness target |
Use Data Catalog views for governed logical access
A Data Catalog view is a definer view. The definer role requires unfiltered Lake Formation SELECT with grant option on every referenced table and column, and CREATE_TABLE on the target database. When a consumer queries the view, the query engine uses the definer role's permissions to access the source tables. You can therefore grant the consumer SELECT on the view without granting direct access to those tables.
Data Catalog views are multi-dialect. One catalog object stores a common schema and metadata together with engine-specific SQL dialects. Each dialect must reference the same tables, columns, and data types. Current supported creation environments include Athena engine version 3, Amazon Redshift, Spark on EMR, and AWS Glue 5.0 or later. A view can reference up to 10 source tables.
Use a Data Catalog view when you need to:
- Hide source tables from consumers.
- Expose selected columns or rows through a reusable SQL definition.
- Apply named-resource or LF-Tag-based permissions to a shared catalog object.
- Maintain one view schema with compatible dialects for supported query engines.
- Return results based on current source data without managing a refresh schedule.
Because the SQL runs when the view is queried, complex joins and aggregations still consume query-engine resources each time. Data Catalog views are therefore a governance and abstraction mechanism, not a query-result cache.
Lake Formation data cell filters can't be attached to a Data Catalog view, and the view definer must have unfiltered, full grantable SELECT on every source table. Put consumer row and column shaping in the view SQL and grant access to the resulting view object. Avoid combining multiple features of Lake Formation when a simpler, auditable view-and-grant design meets the requirement. Review Data filtering limitations and Data Catalog view considerations before choosing this model.
Use Iceberg materialized views for repeated analytical queries
An Apache Iceberg materialized view in the AWS Glue Data Catalog stores the result of its SQL query as a managed Iceberg table. When you configure automatic refresh, the Data Catalog monitors source-table changes and uses managed Spark compute to refresh the stored result. You can also refresh it manually. Refresh can be incremental when the view definition and source changes support it.
The materialized view creating role must have full access to every source table without row, column, or cell filters and access to create table on the target database and location. You will create an execution role for AWS Glue to assume for automatic refresh jobs. Consumers can receive SELECT on the materialized view without receiving access to its source tables.
Use an Iceberg materialized view when you need to:
- Accelerate frequently repeated joins, aggregations, filters, or window functions.
- Reduce repeated scans of large Iceberg tables.
- Replace custom change-detection and refresh orchestration with a managed SQL-based transformation.
- Store precomputed results in a general purpose S3 bucket or an S3 Tables bucket.
- Make a curated result available to Athena SQL, Amazon Redshift, AWS-managed Spark, or another compatible Iceberg reader.
Materialized view DDL and automatic query rewrite require an AWS optimized Spark runtime that uses Spark 3.5.6 or later in Athena for Apache Spark, Amazon EMR, or AWS Glue 5.1. Open-source Spark doesn't support these capabilities.
Automatic query rewrite is opted-in through spark.sql.optimizer.answerQueriesWithMVs.enabled=true. A supported Spark runtime can then use a current materialized view instead of recomputing a matching query from source tables. If the materialized view is stale or can't answer the query accurately, Spark runs the original query against the source tables. Consumers can also query the stored Iceberg table directly from supported SQL engines and compatible Iceberg engines.
This performance benefit introduces two trade-offs:
- Freshness: Materialized views are eventually consistent. The minimum automatic refresh interval is one hour. You can run
REFRESH MATERIALIZED VIEWwhen you need an on-demand refresh, but applications must still account for refresh duration. - Cost: You pay for the Iceberg result storage and the managed compute used for refresh. Compare this cost with the query compute saved by avoiding repeated processing of the source data.
Materialized views work best for stable, high-value query patterns over large Iceberg datasets. They are less suitable when every query must observe the latest source change or when query patterns vary enough that precomputation has limited reuse.
Which option should you choose?
Choose a Data Catalog view when the governing question is, "What rows, columns, and source-table details should this consumer see?" It provides a logical security boundary without storing another copy of the results.
Choose an Iceberg materialized view when the performance question is, "Are we repeatedly running the same expensive transformation, and can we accept a defined refresh interval?" It exchanges storage and refresh compute for lower query latency and less repeated processing.
These features are not interchangeable:
- A logical view prioritizes current source results and governed abstraction.
- A materialized view prioritizes reusable computation and predictable analytical performance.
For enterprise designs, keep each object aligned to one clear purpose. Use Lake Formation grants and LF-Tags to share the selected object, document the definer role, and monitor dependencies. For materialized views, also monitor refresh status, duration, failures, and staleness in Amazon CloudWatch.
Conclusion
Use AWS Glue Data Catalog views to provide governed, multi-engine logical interfaces without exposing source tables. Use Apache Iceberg materialized views to precompute recurring transformations and accelerate latency-sensitive analytical workloads.
Base the decision on four factors: source-table format, freshness requirement, query repetition, and total cost. If governance and current results are the priority, start with a Data Catalog view. If repeated computation is the bottleneck and hourly or manually managed freshness is acceptable, use an Iceberg materialized view.
Resources
- Building AWS Glue Data Catalog views
- Working with Glue catalog views in AWS Glue
- Data Catalog views considerations and limitations
- Iceberg Materialized views with AWS Lake Formation
- Using materialized views with AWS Glue
- Using materialized views with Amazon EMR
- Data filtering and cell-level security in Lake Formation
- Topics
- Analytics
- Language
- English
Relevant content
- Accepted Answer
asked a year ago
asked 2 years ago
AWS OFFICIALUpdated 2 years ago