How to troubleshoot OpenSearch upgrade eligibility failure

Question

I'm trying to migrate an OpenSearch 2.9 cluster to OpenSearch 3.1 but the eligibility test informs the following message:

> The following indices are not compatible with the target Elasticsearch version OS_3.1: [...]. Please refer to our documentation https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-version-migration.html for tips to address this issue

The settings of those indexes are as follows
```
"settings": {
      "index": {
        "number_of_shards": "1",
        "blocks": {
          "read_only_allow_delete": "false",
          "write": "false"
        },
        "provided_name": "index",
        "creation_date": "1655313075513",
        "analysis": {
          "filter": {
            "portuguese_stemmer": {
              "type": "stemmer",
              "language": "light_portuguese"
            },
            "portuguese_stop": {
              "type": "stop",
              "stopwords": "_portuguese_"
            }
          },
          "char_filter": {
            "removeLeadingZeroes": {
              "pattern": "0*([0-9]*)",
              "type": "pattern_replace",
              "replacement": "$1"
            },
            "splitDots": {
              "pattern": """\.""",
              "type": "pattern_replace",
              "replacement": " "
            }
          },
          "normalizer": {
            "Normalizer": {
              "filter": [
                "lowercase",
                "asciifolding"
              ],
              "type": "custom",
              "char_filter": []
            }
          },
          "analyzer": {
            "Analyzer": {
              "filter": [
                "lowercase",
                "portuguese_stop",
                "portuguese_stemmer"
              ],
              "char_filter": [
                "removeLeadingZeroes",
                "splitDots"
              ],
              "tokenizer": "standard"
            }
          }
        },
        "number_of_replicas": "0",
        "uuid": "...",
        "version": {
          "created": "7100299",
          "upgraded": "136407827"
        }
      }
    }
  }
```

so as you can see it does no make use of any of the removed settings:
- `thread_pool.test.max_queue_size`
- `thread_pool.test.min_queue_size`
- `index.store.hybrid.mmap.extensions`
- `index.knn.algo_param.ef_construction`
- `index.knn.algo_param.m`
- `index.knn.space_type`
- `knn.plugin.enabled`

So, I could not understand how specifically those two indexes are at fault... I also tried checking the logs tab but it would state the same error with no more explanation.

How can I troubleshoot it further? Can I have aws give me a more detailed log than "The following indices are not compatible"?

re:Post Agent · Answer

When upgrading from OpenSearch 2.9 to OpenSearch 3.1, there are specific upgrade path requirements and potential compatibility issues that need to be addressed.

First, according to the upgrade path requirements, you cannot directly upgrade from OpenSearch 2.9 to OpenSearch 3.1. You must first upgrade to OpenSearch 2.19 before upgrading to OpenSearch 3.1. This is a mandatory intermediate step in the upgrade process.

For troubleshooting the specific incompatibility issues with your indices, there are several approaches you can take:

1. **Check for deprecated index settings**: While your provided settings don't show the explicitly mentioned deprecated settings, there might be other incompatibilities. OpenSearch 3.1 has several breaking changes from earlier versions.

2. **Monitor validation checks**: When you initiate a configuration change or version upgrade, OpenSearch Service performs validation checks to ensure your domain is eligible for an update. If these checks fail, you'll receive notifications about specific issues.

3. **Use the AWS CLI for more details**: You can use the AWS CLI to get more detailed information about the upgrade status:
```
aws opensearch describe-upgrade --domain-name your-domain-name
```

4. **Check cluster health and status**: Examine your cluster's health, shard allocation, and pending tasks:
```
GET _cluster/health?pretty
GET _cat/shards?v
GET _cluster/pending_tasks
```

5. **Review CloudWatch logs**: Look for OpenSearch service logs in CloudWatch under /aws/aes/domains/your-domain-name for more detailed error information.

If you're still unable to identify the specific incompatibility issues, you may need to contact AWS Support for more detailed information about why your indices are considered incompatible. They can provide specific guidance on what needs to be addressed before the upgrade can proceed.

Remember that the upgrade process involves a blue/green deployment, so ensuring all indices are compatible is crucial for a successful migration.  
**Sources**  
[Upgrading Amazon OpenSearch Service domains - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html)  
[Making configuration changes in Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html)  
[OpenSearch Upgrade Stuck at 60% – Urgent Assistance Required | AWS re:Post](https://repost.aws/questions/QUrsCWpZ0KSCKBVvHAIWqqlA/open-search-upgrade-stuck-at-60-urgent-assistance-required)

How to troubleshoot OpenSearch upgrade eligibility failure

Add your answer

Relevant content