Bedrock Knowledgebase Provisioning Issue with OpenSearch Serverless Collection as the Vector Store

We are working on ingesting websites into an OpenSearch Serverless Collection, with a requirement to provision the infrastructure at runtime and destroy it once vector generation is complete. We are using boto3 APIs for provisioning. Below is an outline of our process:

Process Overview:

1. When a website ingestion job starts, it provisions the following resources via boto3:
   • A VECTOR-SEARCH OpenSearch Serverless collection.
   • A vector index within that collection.
   • A Bedrock Knowledgebase with storage configuration pointing to the above OpenSearch collection.
   • A new data source in the knowledgebase, with the incoming URL attached for ingestion.
2. Once the infrastructure is provisioned, the job initiates the Bedrock Knowledgebase Sync.
   • After the sync completes, the vectors are migrated from the OpenSearch Serverless collection to our internal PostgreSQL database.
3. Finally, the job tears down the infrastructure.

Issue:

We are encountering an error during the Bedrock Knowledgebase provisioning step. The OpenSearch Serverless collection and its index are created successfully, but the create_knowledge_base API call fails with the following error:

An error occurred (ValidationException) when calling the CreateKnowledgeBase operation: The knowledge base storage configuration provided is invalid... no such index [index-name]

It appears that the Bedrock service is unable to locate the newly created index. We have added like a 60sec delay so that index is available before knowledgebase creation but it didn't work. Then we added a delay of like 10minutes but still no luck.

We have used the OpenSearch CAT API to list the created indices, and while the index appears in the list, its health status is empty. We are unsure if this is related to the issue.

Request for Help:

We need assistance in identifying why Bedrock is unable to find the newly created index. Is this something related to the index propagation to the opensearch? or maybe we are missing somthing in the settings. Below, I will share the settings we are using for the relevant boto3 client API calls, as well as the OpenSearch dashboard settings, for better context.

Settings:

We have default 10OCUs for Index and 10OCUs for search set in our Opensearch severless dashboard.

Boto3 Opensearch-serverless client

create_collection (name='some-name', description='some-desc', type='VECTORSEARCH', standbyReplicas='DISABLED' )

Opensearch python client for index creation

    def __load_aws_auth(self):
        service = "aoss"
        credentials = boto3.Session().get_credentials()
        awsauth = AWS4Auth(
            credentials.access_key,
            credentials.secret_key,
            self.region,
            service,
            session_token=credentials.token,
        )

        return awsauth

    def get_opensearch_client(self, collection_endpoint):
        client = OpenSearch(
            hosts=[{"host": collection_endpoint, "port": 443}],
            http_auth=self.__load_aws_auth(),
            use_ssl=True,
            verify_certs=True,
            connection_class=RequestsHttpConnection,
            timeout=300,
        )
        return client

# CREATE VECTOR INDEX
client = get_opensearch_client(collection_endpoint)

index_body = {
         "index":{
            "knn":true,
            "knn.algo_param.ef_search":512
         }
      },
      "mappings":{
         "properties":{
            "bedrock-knowledge-base-vector":{
               "type":"knn_vector",
               "dimension":1024,
               "method":{
                  "engine":"faiss",
                  "name":"hnsw",
                  "parameters":{
                     
                  },
                  "space_type":"l2"
               }
            },
            "AMAZON_BEDROCK_METADATA":{
               "type":"text",
               "index":"false"
            },
            "AMAZON_BEDROCK_TEXT_CHUNK":{
               "type":"text",
               "index":"true"
            }
         }
      }
   }

response = client.indices.create(
    index_name='some-vector-index-name',
    body=index_body,
)

Boto3 Bedrock agent client

create_knowledge_base(
    name='some-name',
    description='some-desc',
    roleArn='bedrock-role-arn',
    knowledgeBaseConfiguration={
      "type":"VECTOR",
      "vectorKnowledgeBaseConfiguration":{
         "embeddingModelArn":"arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0",
         "embeddingModelConfiguration":{
            "bedrockEmbeddingModelConfiguration":{
               "dimensions":1024
            }
         }
      },
    storageConfiguration={
      "type":"OPENSEARCH_SERVERLESS",
      "opensearchServerlessConfiguration":{
         "collectionArn":"",
         "fieldMapping":{
            "metadataField":"AMAZON_BEDROCK_METADATA",
            "textField":"AMAZON_BEDROCK_TEXT_CHUNK",
            "vectorField":"bedrock-knowledge-base-vector"
         },
         "vectorIndexName":"some-vector-index-name"
      }
   })