How do I create an index pattern in my OpenSearch Service cluster?

Resolution

Prerequisites:

• The AWS Identity and Access Management (IAM) user must have PUT and POST permissions to create an index pattern. Example access policy:

  {  "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
          "es:ESHttpHead",
          "es:ESHttpPost",
          "es:ESHttpGet",
          "es:ESHttpDelete",
          "es:ESHttpPut"
        ],
        "Resource": "arn:aws:es:region:account-id:domain/domain-name/*"
      }
    ]
  }

  Note: Replace region with your AWS Region, account-id with your AWS account, and domain-name with your domain name.
• Your cluster version must allow index patterns.

Create the index pattern

Use OpenSearch Dashboards

You can use OpenSearch Dashboards to create an index pattern for OpenSearch Service or Elasticsearch clusters with or without fine-grained access control. For instructions, see Creating an index pattern on the OpenSearch website.

Use curl commands

To create an index pattern for clusters without fine-grained access control, run the following command based on the cluster type.

Elasticsearch clusters:

curl -X POST  https://elasticsearch-end-point/_plugin/kibana/api/saved_objects/index-pattern/ \
-H "kbn-xsrf: true" \

-H "content-type: application/json" \

-d '{ "attributes": { "title": "sample-index*" } }'

Note: Replace sample-index with your index name or pattern.

OpenSearch Service clusters:

curl -X POST  https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/ \
-H "kbn-xsrf: true" \

-H "content-type: application/json" \

-d '{ "attributes": { "title": "sample-index*" } }'

Note: Replace sample-index with your index name or pattern.

For clusters with fine-grained access control, complete the following steps:

1. To generate authorization cookies in the auth.txt file, run the following command based on the cluster type.
   Elasticsearch clusters:

   curl -X POST  https://elasticsearch-end-point/_plugin/kibana/auth/login  \
   -H "kbn-xsrf: true" \
   -H "content-type: application/json" \
   -d '{"username":"usernameexample", "password":"passwordexample"}' \
   -c auth.txt

   Note: Replace usernameexample with your username and passwordexample with your password.
   OpenSearch Service clusters:

   curl -X POST  https://opensearch-end-point/_dashboards/auth/login  \
   -H "kbn-xsrf: true" \
   -H "content-type: application/json" \
   -d '{"username":"usernameexample", "password":"passwordexample"}' \
   -c auth.txt

   Note: Replace usernameexample with your username and passwordexample with your password.
2. To submit the index pattern creation request, run the following command based on your cluster type:
   Elasticsearch clusters:

   curl -X POST  https://elasticsearch-end-point/_plugin/kibana/api/saved_objects/index-pattern/test  \
   -H "kbn-xsrf: true" \
   -H "content-type: application/json" \
   -d '{ "attributes": { "title": "sample-index*" } }' \
   -b auth.txt

   Note: Replace sample-index with your index name or pattern.
   OpenSearch Service clusters:

   curl -X POST  https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/  \
   -H "kbn-xsrf: true" \
   -H "content-type: application/json" \
   -d '{ "attributes": { "title": "sample-index*" } }' \
   -b auth.txt

   Note: Replace sample-index with your index name or pattern.

Use Python

Prerequisites:

• Map the role that runs the Python code to the backend role for fine-grained access control clusters.
• Run the following commands to install the required dependencies:

  pip install boto3
  pip install opensearch-py
  pip install requests
  pip install requests-aws4auth

Run the following Python command to create the index pattern for OpenSearch Service clusters:

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://domain-endpoint/' # include trailing /
region = 'aos-region' # example us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)


path = '_dashboards/api/saved_objects/index-pattern' # _plugin/kibana/api/saved_objects/index-pattern for es versions
url = host + path
payload = {"attributes":{"title":"multi-logs-*","fields":"[]"}}
headers = {"Content-Type": "application/json", "osd-xsrf": "true", "security_tenant": "global" }
r = requests.post (url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)

Note: Replace domain-endpoint with your domain endpoint, and aos-region with your Region. For Elasticsearch clusters, replace _dashboards/api/saved_objects/index-pattern with _plugin/kibana/api/saved_objects/index-pattern.

Troubleshoot index pattern creation issues

You use fine-grained access control with SAML 2.0 or Amazon Cognito authentication

If the domain for your cluster uses SAML 2.0 or Amazon Cognito for authentication, then create an internal user to manage the index pattern.

Note: For clusters where you activated fine-grained access control, the user must have ESHttpPut and ESHttpPost permissions to create an index pattern.

You can't create the index pattern in the Global tenant

By default, OpenSearch Dashboards creates index patterns under the Global tenant. To create an index pattern outside of the Global tenant, run the following command:

curl -s -X POST https://opensearch-end-point/_dashboards/api/saved_objects/index-pattern/sample-index -d '{"attributes": {"title": "sample-index*"}}' \
-H "osd-xsrf:true" \
-H "securitytenant: private" \
-H "content-type:application/json" \
-b auth.txt

Note: Replace sample-index with your index name or pattern.

You didn't include the .kibana alias in the cluster

To troubleshoot this issue, complete the following steps:

1. To check whether the .kibana alias exists in the cluster, run the following command:

   curl -XGET https://opensearch-end-point/_cat/aliases

   Note: For clusters with fine-grained access control, include the -u flag with your username and password. Example command:

   curl -XPOST -u 'master-user:master-user-password' 'domain-endpoint/_cat/indices

   If the .kibana index doesn't exist, then proceed to step 4.
2. To create a backup of .kibana index, run the following command:

   curl -XPOST "https://domain-endpoint/_reindex" -H 'Content-Type: application/json' -d'{
     "source": {
       "index": ".kibana"
     },
     "dest": {
    "index": ".kibana_backup"
     }
   }'

   Note: Replace domain-endpoint with your domain endpoint. For clusters with fine-grained access control, include the -u flag with your username and password.
3. To delete the .kibana index, run the following command:

   curl -XDELETE "https://domain-endpoint/.kibana"

   Note: Replace domain-endpoint with your domain endpoint. For clusters with fine-grained access control, include the -u flag with your username and password.
4. To create a .kibana alias and point it to the .kibana_backup index, run the following command:

   curl -XPOST "https://domain-endpoint/_aliases" -H 'Content-Type: application/json' -d'{
     "actions": [
       {
         "add": {
           "index": ".kibana_backup",
           "alias": ".kibana"
         }
       }
     ]
   }'

   Note: Replace domain-endpoint with your domain endpoint. For clusters with fine-grained access control, include the -u flag with your username and password.

Related information

Export and import Kibana dashboards with OpenSearch Service

Why does the rollover index action in my ISM policy keep failing in OpenSearch Service?