Opensearch Serverless - Slow and fails to index

Question

I created a simple index with the following schema (in a search collection):
```
PUT musers1
{
    "mappings": {
        "properties": {
            "uuid":            { "type" : "keyword" },
            "ext_id":          { "type" : "keyword" },
            "last_name":       { "type" : "text", "fields": { "raw": { "type":  "keyword" }} },
            "first_name":      { "type" : "text", "fields": { "raw": { "type":  "keyword" }} },
            "primary_location":{ "type" : "long" },
            "created_date":    { "type" : "long" },
            "permissions":     { "type" : "keyword" },
            "roles":           { "type" : "keyword" }
        }
    }
}
```
Then I attempted to load 73 records in the dashboard (subset of data):
```
POST _bulk
{"index":{"_index":"musers1","_id":"14399575-cb41-4d33-4ee2-9a69c90a18af"}}
{"created_date":1464369890,"ext_id":"1111111","first_name":"Demo","last_name":"User1","primary_location":481549029373652992,"roles":["MEMBER"],"uuid":"14399575-cb41-4d33-4ee2-9a69c90a18af"}
{"index":{"_index":"musers1","_id":"f9644fb8-e790-429a-4965-9e18be4b8d8f"}}
{"created_date":1464369890,"ext_id":"2222222","first_name":"Demo","last_name":"User2","primary_location":481549029373652992,"roles":["MEMBER"],"uuid":"f9644fb8-e790-429a-4965-9e18be4b8d8f"}
{"index":{"_index":"musers1","_id":"4a621fc9-dc93-4934-756d-450412ca2f7a"}}
{"created_date":1607376361,"ext_id":"63844","first_name":"Demo","last_name":"User3","primary_location":1183110257241493504,"roles":["MEMBER","FT-FLOOR"],"uuid":"4a621fc9-dc93-4934-756d-450412ca2f7a"}
...
```
And I got back the response 4 minutes later!
```
{
  "took": 251331,
  "errors": true,
  "items": [
...
```
In the end, there were 64 records indexed, and the other nine failed with the error `Timeout while processing request`.

Also, I loaded 2000 records (slowly and painfully) yesterday into a different index and ran a few searches against it. I ended up with 5 Search and 5 Index OCUs. These OCUs have remained since yesterday. Is it normal to have 10 OCUs active when there is no indexing or search activity?

Is it normal for 73 records to fail 12% of the time and take 4 minutes? That's 18 records per minute. If I had 20 million records to index, (I have more than that actually) it would take over 2 years to index!

I hope I am doing something wrong, because this seems too slow and too expensive...

Any ideas?
Z

Answer

When you create your first collection, OpenSearch Serverless instantiates two OCUs—one for indexing and one for search. These OCUs exist even when there's no activity on any collection endpoints. All subsequent collections share these OCUs. When you create additional collections in the same account, OpenSearch Serverless only adds additional OCUs for search and ingest as needed to support the collections, according to the capacity limits that you specify. Capacity does scale back down as your compute usage decreases. [1]

If you are facing issue with OCUs not scaling down and latency while ingesting records, recommend you to kindly open a case with AWS Premium Support team so that they can have a better look at the resources.

[1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html#serverless-process

Opensearch Serverless - Slow and fails to index

Relevant content