OpenSearch slow returning a big result set

0

I have this query:

GET my-index/_search
{
    "query": {
        "range": {
            "_lastUpdated": {
                "gt": 1723262400999
            }
        }
    },
    "size": 10000,
    "sort": [
        {
            "@rid": {
                "order": "asc"
            }
        }
    ],
    "_source": false
}

executed against an index with 13 million documents of which around 5 million match takes ~4 seconds, that seems a lot to me.

The index size is ~3 GiB, with 5 shards and 1 replica.

There is no other activity on the cluster.

Cluster has a single t3.medium.search node, with EBS volume with 3,000 IOPS and 125 MiB/s throughput.

What I tried:

  • double EBS IOPS and throughput - no appreciable difference

  • double EBS IOS and throughput and switch to a m6g.large.search node - somewhat better but still slow (maybe ~ 3 seconds)

  • double EBS IOS and throughput, switch to a m6g.large.search node, switch to an index with a single shard and replica - better but still slow (~ 1.5 seconds)

  • go back to original hardware but with 5 nodes instead of 1 - same performance as the baseline (~4 seconds)

  • try a term query instead of range - same speed.

2 Answers
1

Hi,

You may be interested in this very recent blog post to optimize your search queries: https://aws.amazon.com/blogs/big-data/improve-your-amazon-opensearch-service-performance-with-opensearch-optimized-instances/

The final sections give you numbers to compare with your own.

Also if you go on the OpenSearch benchmark leaderboard, you'll get multiple results to compare with: https://opensearch.org/benchmarks/

Best,

Didier

profile pictureAWS
EXPERT
answered 2 months ago
0

Thanks Didier, that's helpful

I did some more experiments, the issue (not surprisingly) is the sort, and switching to a sorted index on @rid reduces the time to ~ 0.5 seconds, even with the single t3.medium.search node.

mimo
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions