How to build a cross-account data mesh with Amazon S3 Tables and AWS Lake Formation

Amazon S3 Tables provide managed Apache Iceberg tables stored in purpose-built S3 table buckets. By combining S3 Tables with AWS Lake Formation, you can build a cross-account data mesh where a producer account owns the data, a central governance account owns the catalog and governs access, and one or more consumer accounts query the shared tables — all without copying data or managing cross-account S3 bucket policies for every consumer.

This article walks through an end-to-end setup using three accounts:

• Producer account – owns the S3 table bucket and the underlying table data.
• Central governance account – registers the producer's table bucket with Lake Formation, creates a federated Glue catalog over it, and governs who can access what.
• Consumer account – receives a Lake Formation share and queries the tables with Amazon Athena and Apache Spark.

The example account IDs below are placeholders. Replace them with your own:

• Producer: 111111111111
• Central governance: 222222222222
• Consumer: 333333333333

All three accounts in this example are in us-west-1.

Reference documentation

• Registering an S3 table bucket in another account
• Accessing shared S3 tables with Lake Formation

Prerequisites

• Three AWS accounts (producer, central governance, consumer) or adapt to your account topology. For the producer, central and consumer account actions in this article, it is recommended to use the Lake Formation administrator role.
• An S3 table bucket in the producer account. For example, producer1tablebucket.
• An IAM role in the central governance account that Lake Formation will assume to access the producer's table bucket (this article uses S3TableAccessToLakeFormation_custom). Permissions on this role's IAM policy and its trust policy and Lake Formation permissions are shown in Prerequisites for integrating Amazon S3 tables catalog with the Data Catalog and Lake Formation.
• AWS Command Line Interface (AWS CLI) configured for each account. Several of the steps below are CLI-only because the equivalent console options are not yet available.
• Lake Formation administrator access in the central governance and consumer accounts.

Step 1 — Producer account: grant the central account access to the table bucket

In the producer account, attach a resource policy to the S3 table bucket that allows the central account's Lake Formation role to access the bucket and its tables.

aws s3tables put-table-bucket-policy \
    --table-bucket-arn arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket \
    --resource-policy '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::222222222222:role/S3TableAccessToLakeFormation_custom"
            },
            "Action": [
                "s3tables:*"
            ],
            "Resource": [
                "arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket/table/*",
                "arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket"
            ]
        }
    ]
}'

Verify the policy:

aws s3tables get-table-bucket-policy \
    --table-bucket-arn arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket

Note: Scope the Action and Resource down to least privilege for your environment. s3tables:* is shown here for brevity.

Step 2 — Central governance account: register the bucket and create a federated catalog

We will register the S3 table bucket with Lake Formation, so that Lake Formation can manage the table bucket access and enable sharing with consumer accounts.

2a. Register the producer's table bucket with Lake Formation

aws lakeformation register-resource \
    --resource-arn 'arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket' \
    --role-arn arn:aws:iam::222222222222:role/S3TableAccessToLakeFormation_custom \
    --region us-west-1

2b. Create a federated Glue catalog over the table bucket

We will make the registered S3 table bucket available as a Glue catalog s3tablescatalog-central in the central account. Since the catalog will point to an S3 table bucket, instead of standard S3, it will be referred to as a federated catalog (not Glue default catalog).

aws glue create-catalog --region us-west-1 \
    --cli-input-json '{
   "Name": "s3tablescatalog-central",
   "CatalogInput": {
      "FederatedCatalog": {
         "Identifier": "arn:aws:s3tables:us-west-1:111111111111:bucket/producer1tablebucket",
         "ConnectionName": "aws:s3tables"
      },
      "CreateDatabaseDefaultPermissions": [],
      "CreateTableDefaultPermissions": []
   }
}'

Note: After creating the catalog, the Lake Formation administrator may not automatically have permissions on it. Self-grant SUPER (or the specific permissions you need) on the new catalog before continuing.

2c. Create a database (namespace) in the federated catalog

aws glue create-database \
    --region us-west-1 \
    --catalog-id "222222222222:s3tablescatalog-central" \
    --database-input '{ "Name": "salesdatanamespacecentral" }'

2d. Verify the catalog and database

aws glue get-catalog --catalog-id "222222222222:s3tablescatalog-central"

aws glue get-databases --catalog-id "222222222222:s3tablescatalog-central"

You should see the federated catalog pointing at the producer's table bucket and your new database listed.

2e. Create a table in the namespace

aws glue create-table \
    --database-name salesdatanamespacecentral \
    --catalog-id 222222222222:s3tablescatalog-central \
    --region us-west-1 \
    --table-input '{
        "Name": "dailysalescentral",
        "Parameters": { "format": "ICEBERG" },
        "StorageDescriptor": {
           "Columns": [
              {"Name": "sales_amount", "Type": "double", "Parameters": {"required": "true"}}
           ]
        }
}'

Confirm the table was created:

aws glue get-tables \
    --database-name salesdatanamespacecentral \
    --catalog-id 222222222222:s3tablescatalog-central

The returned table is an Iceberg table backed by the producer's S3 table bucket (its Location and s3TableArn point back to the producer-owned storage).

Step 3 — Producer account: load data with Athena

Back in the producer account, use Athena (or Spark) to insert data into the tables created in the federated namespaces. Because the storage is owned by the producer, the producer can write to these tables directly. For example:

INSERT INTO dailysalescentral VALUES (1234.56);

Step 4 — Central governance account: share with the consumer

Grant Lake Formation permissions on the database/table to the consumer account. For example, share salesdatanamespacecentral.dailysalescentral from the central catalog to the consumer account using either the named-resource method or LF-Tags, just as you would share a standard Lake Formation table. This creates an AWS Resource Access Manager (AWS RAM) share to the consumer account.

Step 5 — Consumer account: accept the share and query

1. Accept the AWS RAM share invitation from the central governance account. Once accepted, the shared S3 Tables catalog namespace appears under Databases in Lake Formation.
2. Create a resource link to the shared database. For example, rl_salesdatanamespace.
3. (Optional) Share the resource link with additional roles in the consumer account so they can query it.
4. Query the resource link. Standard access works in both Athena and Spark.

Query with Athena

Run a standard SELECT against the resource-link database in the Athena query editor.

SELECT * FROM rl_salesdatanamespace.dailysalescentral LIMIT 10;

Query with Apache Spark (AWS Glue/EMR)

import logging
import sys
from pyspark.sql import SparkSession

# Logging configuration
formatter = logging.Formatter('[%(asctime)s] %(levelname)s @ line %(lineno)d: %(message)s')
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(handler)

aws_region = "us-west-1"
aws_account_id = "333333333333"          # consumer account
local_catalogname = "spark_catalog"
warehouse_path = "file:///tmp/spark-warehouse"

localcatalog_database = "rl_salesdatanamespace"   # resource-link database
localcatalog_table = "dailysalescentral"

spark = (SparkSession.builder.appName('s3tables-datamesh')
    .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
    .config(f'spark.sql.catalog.{local_catalogname}', 'org.apache.iceberg.spark.SparkCatalog')
    .config(f'spark.sql.catalog.{local_catalogname}.catalog-impl', 'org.apache.iceberg.aws.glue.GlueCatalog')
    .config(f'spark.sql.catalog.{local_catalogname}.client.region', aws_region)
    .config(f'spark.sql.catalog.{local_catalogname}.glue.account-id', aws_account_id)
    .config(f'spark.sql.catalog.{local_catalogname}.io-impl', 'org.apache.iceberg.aws.s3.S3FileIO')
    .config(f'spark.sql.catalog.{local_catalogname}.warehouse', warehouse_path)
    .getOrCreate())

logger.info("Spark session started successfully")

# List databases in the default catalog
spark.sql("SHOW DATABASES").show()

# List tables and read from the shared table through the resource link
spark.sql(f"SHOW TABLES IN {local_catalogname}.{localcatalog_database}").show()
spark.sql(f"SELECT * FROM {local_catalogname}.{localcatalog_database}.{localcatalog_table} LIMIT 5").show()

logger.info("Spark job completed successfully")

Summary

In this article you built a cross-account data mesh on Amazon S3 Tables governed by AWS Lake Formation:

• The producer authorized the central account's Lake Formation role on its S3 table bucket.
• The central governance account registered the bucket, created a federated Glue catalog and namespaces, and created Iceberg tables over the producer's storage.
• The producer loaded data with Athena.
• The central account shared a table with a consumer, which created a resource link and queried it with both Athena and Spark.

This pattern lets you share a governed subset of data across accounts without duplicating data or hand-maintaining S3 bucket policies per consumer. Try it in your environment and adapt the IAM and Lake Formation permissions to your own least-privilege requirements.