Best Practices for Using Apache Iceberg v3 Tables in Amazon Redshift

Apache Iceberg is an open table format for data lakes that provides ACID transactions and schema evolution. Iceberg v3 builds on v2 with three key features: default column values that simplify schema evolution without rewriting data, row lineage pseudo-columns that enable incremental pipelines and CDC (change data capture) workflows, and deletion vectors that replace positional delete files with compact compressed bitmaps for faster reads and writes. Amazon Redshift supports all three features on Graviton-based provisioned clusters (RG) and Redshift Serverless workgroups.

Technical Prerequisites Before working with Iceberg v3 tables, ensure you have an Amazon Redshift RG provisioned cluster or Serverless workgroup with an associated IAM role that permits access to your data lake (S3 and AWS Glue Data Catalog). An external schema referencing your Glue Data Catalog database must be configured. Generate column statistics for your Iceberg tables using AWS Glue to ensure optimal query planning — Redshift uses these statistics to reduce file scans during query processing.

Creating Iceberg v3 Tables To create a new Iceberg v3 table, specify the format version in the TABLE PROPERTIES clause:

CREATE TABLE my_schema.orders (
  id INT,
  customer_id INT,
  status VARCHAR DEFAULT 'pending',
  priority INT DEFAULT 0,
  order_date TIMESTAMPTZ
)
USING ICEBERG
LOCATION 's3://my-data-lake/orders/'
TABLE PROPERTIES ('format-version' = '3');

If you omit format-version, Amazon Redshift creates the table as Iceberg v2 by default. For new tables where you plan to use default values, row lineage, or high-frequency updates/deletes, start with v3 from the beginning.

Upgrading from v2 to v3 You can upgrade existing v2 tables in place with a single metadata-only operation — no data rewrite is required:

ALTER TABLE my_schema.orders
SET TABLE PROPERTIES ('format-version' = '3');

Keep these considerations in mind when upgrading:

• No downgrade path — once upgraded to v3, you cannot revert to v2. Test thoroughly in a non-production environment first.

• Row lineage is not immediate — pre-upgrade data returns NULL for _row_id and _last_updated_sequence_number until the first write operation after the upgrade, which generates lineage values for the entire table.

• Existing v2 delete files remain valid — they are applied during reads. Subsequent DELETE, UPDATE, or MERGE operations automatically merge v2 positional deletes into deletion vectors.

• Timestamptz mapping changes — in v3, the Iceberg timestamptz type maps to the Redshift TIMESTAMPTZ type (timezone-aware), whereas v2 mapped it to TIMESTAMP. Validate any downstream consumers that depend on timestamp behavior.

Leveraging Default Column Values Default column values simplify schema evolution by allowing you to add new columns without rewriting existing data files. When you add a column with a default, reads of previously written data return the default automatically:

ALTER TABLE my_schema.orders
ADD COLUMN region VARCHAR DEFAULT 'us-east-1';

Best practices for default values:

• Use defaults for columns that genuinely have a meaningful initial state (e.g., status fields, region identifiers, feature flags). Avoid defaulting to NULL when a concrete value would be more useful.

• Remember that CTAS does not inherit defaults from the source table — use ALTER TABLE ALTER COLUMN SET DEFAULT after creation if needed.

• When changing a default with SET DEFAULT, the new value applies only to files written after the change. Existing files retain the previous default. Plan your schema changes accordingly.

• Only literal values are supported — expressions, functions, and nested data types cannot be used as defaults.

Building Incremental Pipelines with Row Lineage Row lineage exposes two pseudo-columns — _row_id (unique row identifier) and _last_updated_sequence_number (snapshot sequence number of the last modification). These enable efficient CDC and incremental processing:

-- Find all rows modified since snapshot sequence 42
SELECT _row_id, _last_updated_sequence_number, *
FROM my_schema.orders
WHERE _last_updated_sequence_number >= 42;

Best practices for row lineage:

• Row lineage columns are **not included in SELECT ***. You must name them explicitly in your SELECT list.

• Use _last_updated_sequence_number as a high-water mark in incremental pipelines. Record the max sequence number after each pipeline run, then filter for rows above that threshold on the next run.

• For tables upgraded from v2, the first write operation after the upgrade generates row lineage values for the entire table. Ensure a write has occurred before building pipelines that depend on lineage pseudo-columns.

• Row lineage is ideal for feeding downstream systems — materialized views, aggregation tables, or external consumers — with only the rows that changed since the last sync.

Optimizing Performance with Deletion Vectors Deletion vectors replace v2 positional delete files with compressed bitmaps, delivering faster reads and writes for high-frequency update/delete workloads:

• Automatic — no SQL syntax changes required. DELETE, UPDATE, and MERGE operations on v3 tables produce deletion vectors instead of positional delete files.

• Self-compacting — subsequent deletes on the same data file produce a new deletion vector that merges previous deletions, maintaining a single deletion vector per data file. This prevents the accumulation of many small delete files that degrade read performance.

• Compliance-friendly — ideal for workloads that require compliance-driven record removal, where v2 positional delete files would accumulate rapidly and slow reads.

If your workload involves frequent row-level deletes or updates, upgrading to v3 can yield measurable read and write performance improvements simply from the deletion vector format change.

Security Best Practices Amazon Redshift supports AWS Lake Formation fine-grained access control on Apache Iceberg tables, enabling column-level and row-level security. Scope IAM roles to the minimum permissions needed for Glue Data Catalog and S3 access. Note that federated identity is not supported when writing to Iceberg tables — avoid using the SESSION keyword for the IAM_ROLE parameter when creating external schemas used for Iceberg writes. You can also use user-defined data handling parameters on Iceberg tables to tailor data queries and avoid scan errors from schema-to-file mismatches. Additionally, Iceberg v3 tables cannot fall back to positional delete files for new write operations. While existing v2 positional delete files can still be read, all new deletes must use deletion vectors.

Known Limitations Be aware of the current limitations when working with Iceberg v3 in Amazon Redshift:

• Complex types (struct, list, map, variant) are not supported for reading or writing in v3 tables.

• The following data types are also not supported in v3 tables: geometry, geography, binary, uuid, time, timestamp_ns, timestamptz_ns, and unknown.

• Time travel queries are not currently supported with Iceberg tables.

• Downgrading from v3 to v2 is not supported.

• Materialized views on Iceberg tables do not support automatic query rewriting.

Conclusion Apache Iceberg v3 support in Amazon Redshift brings meaningful improvements to schema flexibility, incremental processing, and update/delete performance for data lake workloads. Start new tables on v3 when you need default column values or row lineage, and upgrade existing v2 tables when high-frequency update/delete workloads would benefit from deletion vectors. Test the upgrade path in non-production first, validate timestamp behavior changes, and generate column statistics with AWS Glue to ensure optimal performance.

References For more detailed information:

• Apache Iceberg v3 features in Amazon Redshift

• Using Apache Iceberg tables with Amazon Redshift

• Iceberg Performance Tuning for Amazon Redshift (re:Post)

• Getting started with Apache Iceberg write support in Amazon Redshift

• Amazon Redshift now supports Apache Iceberg v3 tables (What's New)