How do I use Lake Formation to share AWS Glue Data Catalog databases and tables cross-account?

7 minute read

I want to use AWS Lake Formation to share AWS Glue Data Catalog databases and tables across AWS Accounts.

Resolution

With Lake Formation's cross-account feature, you can grant access to other AWS accounts to write and share data to or from the data lake. You can share the resources either through tag-based access control or named resources.

To use cross-account sharing, use one of the following approaches:

Hybrid access mode
Default AWS Identity and Access Management (IAM) Lake Formation permissions mode

Prerequisites:

Add permissions required for cross-account access. For more information, see Sharing Data Catalog tables and databases across AWS accounts or IAM principals from external accounts and AWSLakeFormationCrossAccountManager.
Turn on sharing with AWS Organizations.
Meet cross-account sharing prerequisites.
For Hybrid access mode, update your Cross-account version settings to version 4. For more information, see Common hybrid access mode use cases.
For Default AWS IAM Lake Formation permissions mode, revoke all Lake Formation permissions from the IAMAllowedPrincipals group for the data catalog resource. Also, change the default permissions for Data Catalogs to not have Super permissions. For more information, see Changing the default settings for your data lake.

Note: To use either the hybrid access mode or the default IAM Lake Formation permissions mode, you must share your database and database tables with the target account outside of your organization. When you share all tables in the source account, new tables that you create in the source account are automatically shared with the target account.

Hybrid access mode

In the source account, complete the following tasks.

Register the Amazon S3 location with hybrid access mode

Complete the following steps:

Open the Lake Formation console, and sign in as a data lake administrator.
In the navigation panel, under Administration, choose Data lake locations.
On the Data lake locations page, choose Register location.
On the Register location page, enter the following information:
For Amazon S3 location, provide the Amazon Simple Storage Service (Amazon S3) location of your database and tables.
For IAM role, provide an AWS Identity and Access Management (IAM) role that can access the data in the Amazon S3 location. For more details about this role requirement, see Requirements for roles used to register locations.
For Permission mode, choose Hybrid access mode.
Choose Register location.

After you register the Amazon S3 location, review the location information for the data lake. Verify that the permissions mode is Hybrid access mode.

Grant cross-account permissions for the database and table

Complete the following steps:

Open the Lake Formation console, and sign in as a data lake administrator.
In the navigation pane, choose Database, and then select the database.
Choose Actions, and then choose Grant.
On the Grant permissions screen, under Principals, choose External account.
For AWS account ID or AWS organization ID, enter the account ID of the target account.
Under Hybrid access mode, select Make Lake Formation permissions effective immediately.
Choose Grant.
To grant cross-account permissions for the table, repeat the preceding steps. Make sure to select the All tables or Specific table option.

Note: This configuration opts-in the consumer account Lake Formation administrator roles to use Lake Formation permissions. This update doesn't interrupt access to the consumer account's IAM and S3 access for the same database. This sharing doesn't revoke access to the IAMAllowedPrincipal group.

Default IAM Lake Formation permissions mode

In the source account, complete the following tasks.

Share a database and its tables with the target account

Complete the following steps:

Open the Lake Formation console, and sign in as a data lake administrator.
In the navigation pane, choose Database, and then select the database.
Choose Actions, and then choose Grant.
On the Grant permissions screen, under Principals, choose External account.
For AWS account ID or AWS organization ID, enter the account ID of the target account.
For Table, select All tables.
For Table permissions and Grantable permissions, select the access permissions that you want to grant.
Choose Grant.

Configure your resources

For the target account, the process is the same for both hybrid mode and default IAM Lake Formation permissions mode for the following configurations:

Accepting shared resources
Granting permissions to IAM users or roles
Creating resource links

To configure your resources, complete the following steps:

Accept the resources shared with you. For more information, see Access AWS resources shared with you.
Create a resource link to a shared Data Catalog database. For more information on resource links, see Creating resource links.
Grant IAM users or principals the required permissions for the resource link and shared database.

Note: IAM users can also view the database and resource link in the Amazon Athena console or Amazon Redshift Spectrum. IAM users and principals from the target account must have access to the Amazon S3 path in the source account.

To grant access to the IAM users for the shared databases, complete the following steps:

Open the Lake Formation console, and sign in as a data lake administrator.
In the navigation pane, choose Databases.
Select the shared database.
Choose Actions, and then choose Grant.
Under Principals, select IAM users and roles. Then, select the IAM user or principal that you want to grant access permissions for.
Under Database permissions, select Describe.
Choose Grant.

Note: This step provides the minimum permissions for the users to view the shared database.

To grant access to all tables or to specific tables in the database, select the All tables option:

Under All tables, select the resource link.
Choose Actions, and then choose Grant.
Select IAM users and roles. Then, select the IAM user or principal that you want to grant access permissions for.
Under LF-Tags or catalog resources, do the following:
To grant access to all tables in the database, for Tables - optional, select All tables.
To grant access to only specific tables in the database, for Tables - optional, select the specific tables.
For Table permissions and Grantable permissions, choose Select, and then choose Describe.
Choose Grant.

Note: You can grant only those permissions that you selected for Grantable permissions in the source account.

After you grant the required permissions, you can query the table in Athena from the target account.

Note: If you revoke the permissions that you granted earlier from the source account, then the target account can't access the shared database or table. However, the resource link that you created in the target account isn't automatically deleted. You must manually delete the resource link.

Share only tables with the target account

To share individual tables with the target account, complete the steps in the "Configure your resources" section. As part of the steps, update the following resources:

Source account

To grant access to the target account from the Lake Formation console, select the individual tables instead of selecting the database.

Target account

To access the shared table in the Lake Formation console, accept the resource share in the AWS Resource Access Manager (AWS RAM) console. Then, create a resource link for the shared table. After the resource link is created, you can query the shared table with the data lake administrator access.

To grant access to the IAM users and principals for the shared table, you must grant permissions for the resource link.

Note: When you grant permissions on the table, you can restrict access only for the columns in the table. If you restrict access, then the target account can view only those columns in the shared table.

When you delete a database or table, Lake Formation doesn't automatically delete the resource shares in AWS RAM. Before you delete a shared database or table, you must manually revoke cross-account permissions.