How do I use the Amazon OpenSearch Serverless collection with a private network policy to create a knowledge base in Amazon Bedrock?

Resolution

You can use Amazon OpenSearch Serverless as a vector store within a knowledge base. The embed allows the knowledge base to access those OpenSearch Serverless collections that have a private network policy.

Prerequisites

The example in the following sections has these permissions:
Note: Additional restrictions might be needed for specific use cases.

User or console permissions

• AmazonBedrockFullAccess policy for the AWS Identity and Access Management (IAM) user role.
• AmazonS3FullAccess policy for the IAM use role.
• Inline policy for collections. For more information, see Creating, listing, and deleting Amazon OpenSearch Serverless collections.
• Inline policy for OpenSearch API and Dashboard access. For more information, see Using OpenSearch API operations.
• PassRole that provides permission for your custom service role. For more information, see Granting a user permissions to pass a role to an AWS service.

Roles

• A service role.

Create a collection with a private network policy

Complete these steps:

1. Log in to the Amazon OpenSearch Service console.

2. In the left navigation pane, expand Serverless. Then, choose Collections.

3. Choose Create collection. Enter the collection details.

4. Choose Standard create.

5. Under Network access settings, configure network access for the collection:

   For Access type, select private. Then, select the Amazon Virtual Private Cloud (VPC) endpoints and Amazon Bedrock service that can access the collection.

   For VPC endpoints for access, specify one or more VPC endpoints. Or, select the Create VPC Endpoint option to generate the VPC endpoint when you create the private collection.

   For AWS service private access, select Amazon Bedrock as a service (bedrock.amazonaws.com).

   For Resource type, select Enable access to OpenSearch endpoint.
   Note: AWS service private access applies only to the OpenSearch endpoint, not to the OpenSearch Dashboards endpoint.

6. Choose Next.

7. Configure the data access rules for the collection. The rules define who can access the data within the collection. For each rule that you create, perform the following steps:
   Choose Add principals, and then select one or more AWS Identity and Access Management (IAM) roles.
   Choose Security Assertion Markup Language (SAML) users and groups to provide data access. Make sure to include Amazon Bedrock's custom service role that you need to create the knowledge base.
   Under Grant permissions, select all the alias, template, and index permissions to grant the associated principals.

8. Choose Next.

9. Under Data access policy settings, choose Create as a new data access policy. Or, add the principals and permissions to an existing policy.

10. Review your collection configuration. Then, choose Submit.

Create an index from an OpenSearch Dashboard

You can't directly create an index for a private collection. The following example shows you how to use the OpenSearch Dashboard to create an index and the vector metadata field. Modify the existing network policy or create a new policy to provide access to the OpenSearch Dashboard.

Follow these steps to modify the existing networking policy:

1. Log in to the Amazon OpenSearch Service console.
2. In the left navigation pane, expand Serverless. Then, choose Collections.
3. Go to the network policy section of the private collection.
4. Select your policy in Associated Policy.
5. Choose Edit.
6. Choose Add Rule.
7. Select Public for Access type.
8. Select Enable access to OpenSearch Dashboards.
9. Search and associate the private collection name.
   Note: Use the Update button to update an existing policy.

Create a vector index

Complete the following steps:

1. Log in to the Amazon OpenSearch Service console.

2. In the left navigation pane, expand Serverless. Then, choose Collections.

3. Select the private collection.

4. Select Create vector index.

5. Enter details for Vector index, Vector fields, and Metadata management.

6. Copy the JSON code block with settings.

7. In the left navigation pane, expand Serverless. Return to Collections.

8. Open the Dashboard link to access the OpenSearch Dashboard.

9. From Home on the left, select Developer tools.

10. Enter the JSON code that you copied earlier along with its settings. Use a PUT command as shown below. Then, run the code:

    PUT <index name>
    {
      "settings": {
        "index": {
          "knn": true,
          "knn.algo_param.ef_search": 512
        }
      },
      "mappings": {
        "properties": {
          "<vector field name>": {
            "type": "knn_vector",
            "dimension": 1536,
            "method": {
              "name": "hnsw",
              "engine": "faiss",
              "parameters": {},
              "space_type": "l2"
            }
          },
          "<Mapping field-1>": {
            "type": "text",
            "index": "true"
          },
          "<Mapping field-2>": {
            "type": "text",
            "index": "true"
          }
        }
      }
    }

    To customize the code, replace the following variables:

    <index name>, <vector field name>, <Mapping field-1>, <Mapping field-2>

    For more information, see Create index on the OpenSearch website.
    Note: You can also choose to create your own vector index. For information on how to create a vector index with your code, see Ingesting data into Amazon OpenSearch. For information on vector store expectations from Amazon Bedrock, see Set up a vector index for your knowledge base in a supported vector store.

Create a knowledge base

1. Before you create a knowledge base, make sure that the private collection is ready and available. Verify that the user role and the Amazon Bedrock service roles have data and network access. Confirm that the vector index has the following fields:
   Collection ARN
   Vector index name
   Vector field name
   Metadata management (first mapping field)
   Metadata management (second mapping field)
2. Create the knowledge base.

Troubleshooting tip

Error: "The knowledge base storage configuration provided is invalid... Request failed: [http_exception] server returned 401"

This error occurs when Amazon Bedrock isn't added to the network policy to give it access to the private collection and the VPC endpoint.

To resolve this error, go to step 5 of Create a collection with a private network policy in the earlier section. Then, add bedrock.amazonaws.com to Network Policy to allow Amazon Bedrock access to the private collection.

Related information

How do I troubleshoot the permission errors that I get when I create a knowledge base in Amazon Bedrock?