Creating Knowledge Bases for Structured Data through SageMaker Unified Studio Results in Unknown Error

Based on the information provided, there are a few potential issues that could be causing the "Unknown error" when trying to create a Knowledge Base with Amazon Bedrock using the Lakehouse data source in SageMaker Unified Studio.

First, it's important to note that Knowledge Bases in Amazon Bedrock can use various data sources, including documents (like PDFs), web crawlers, or data sources already in your project (such as Amazon S3 buckets or structured data in Amazon Redshift). However, there may be limitations or specific requirements when using Lakehouse as a data source.

Since you mentioned that creating a Knowledge Base using S3 as the data source works fine, but using the Lakehouse data source fails, this suggests a potential permission issue or integration limitation between Amazon Bedrock and the SageMaker Lakehouse.

Here are some potential solutions to investigate:

1. Permissions: Even if you have administrator access, the Bedrock service itself might need explicit permissions to access your Lakehouse data. Similar to how S3 bucket policies need to explicitly allow Bedrock to read from buckets, there might be additional permissions needed for Lakehouse access.

2. User Profile Configuration: Check if your SageMaker Studio user profile has the necessary permissions to bridge between Bedrock and Lakehouse. SageMaker Studio requires unique user profiles with appropriate permissions to access and launch resources.

3. Data Format Compatibility: Verify that the data format in your Lakehouse tables is compatible with what Bedrock Knowledge Base can ingest. Even though the tables and columns are visible, there might be data type or format issues preventing successful ingestion.

4. Region Availability: While you've tried in both us-east-1 and us-east-2, check if the specific integration between Bedrock Knowledge Bases and SageMaker Lakehouse is fully supported in those regions.

Since there are no logs in CloudTrail, CloudWatch, or CloudFormation, this suggests the error might be occurring at an earlier stage in the process, possibly during the validation of the request rather than during the actual deployment.

You might want to try using the AWS CLI or SDK to create the Knowledge Base with more detailed error reporting, or contact AWS Support for more specific guidance on this integration.
Sources
Add a Knowledge Base to your Amazon Bedrock app - Amazon SageMaker Unified Studio
Bedrock Knowledge Base Creation: Failure to Ingest from S3 | AWS re:Post
Troubleshooting - Modern Data Architecture Accelerator

Hey,

Hope you're keeping well.

This behavior suggests the Lakehouse data source integration in SageMaker Unified Studio isn’t passing the required metadata or credentials through to Bedrock during KB deployment. Since S3 works but Lakehouse fails, check that the Lakehouse project’s Glue catalog database and tables have the correct permissions for the SageMaker execution role, including glue:GetTable, glue:GetDatabase, and any underlying S3 bucket access. Also verify that the Lakehouse data source is in the same region as the Bedrock KB and that the data format is supported for ingestion.

Thanks and regards,
Taz

Hey,

Hope you're keeping well.

It sounds like the Lakehouse data source integration in SageMaker Unified Studio is failing before Bedrock even starts KB deployment. Since S3 works but Lakehouse doesn’t, this usually points to missing permissions or unsupported data formats when Bedrock tries to read from the Lakehouse catalog. I’d confirm that the Lakehouse’s backing Glue Data Catalog tables use supported formats (Parquet/CSV) and that the IAM role for Unified Studio has glue:GetTable, glue:GetTableData, and any required access to the underlying storage location.

Thanks and regards,
Taz