How do I troubleshoot slow logs in Amazon OpenSearch Service?

Resolution

Unable to create the resource access policy error

If your AWS account exceeds 10 resource policies in a single AWS Region, then you receive the following error in Amazon CloudWatch Logs:

"Unable to create the Resource Access Policy - You have reached the maximum limit for number of Resource Access Policies for CloudWatch Logs. Please select an existing policy and edit it or delete an older policy and try again."

To resolve this error, consolidate your AWS Identity and Access Management (IAM) resource policies into one IAM policy, delete the existing IAM policies, and replace them with the new IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "es.amazonaws.com"
            },
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogStream"
            ],
            "Resource": [
                "ARN-Log-Group-1",
                "ARN-Log-Group-2",
                "ARN-Log-Group-3",
                "ARN-Log-Group-4"
            ]
        }
    ]
}

Note: Replace ARN-Log-Group-1, ARN-Log-Group-2, ARN-Log-Group-3, and ARN-Log-Group-4 with your CloudWatch Logs group ARNs.

Important: You can't increase the IAM policy quota.

Slow logs aren't delivered

If you don't see your slow logs delivered to CloudWatch, then update your IAM policy or your OpenSearch Service thresholds.

IAM policy

You must configure your IAM policy to log your queries because OpenSearch Service requires permission to write to CloudWatch Logs.

To update your IAM policy, complete the following steps:

1. Open the OpenSearch Service console.
2. Choose your domain.
3. Choose the Logs tab, and then choose Search Slow Logs.
4. Choose Select Setup.
5. Update your IAM policy with your resource values:

   
   {  "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "es.amazonaws.com"
         },
         "Action": [
           "logs:PutLogEvents",
           "logs:CreateLogStream"
         ],
         "Resource": "arn:aws:region:account_id:log-group:/aws/aes/domains/myes/search-logs:*"
       }
     ]
   }

   Note: Replace region with your Region. Replace account-id with your account Id.
6. (Optional) For additional security, add the aws:SourceAccount and aws:SourceArn condition keys to your IAM policy:

   {
     "Condition": {
       "StringEquals": {
         "aws:SourceAccount": "account-id"
       },
       "ArnLike": {
         "aws:SourceArn": "arn:aws:es:region:account-id:domain/domain-name"
       }
     }
   }

   Note: Replace region with your Region. Replace account-id with your account Id. Replace domain-name with your domain name.
7. Choose Enable.

For more information, see Enable log publishing (console).

OpenSearch thresholds

By default, OpenSearch Service has logging turned off. You can use an API call to set the log level and timing threshold. These settings are dynamic, so you can update the settings on a running cluster, and OpenSearch Service applies them immediately.

Set an appropriate timing threshold for your domain. If all your requests complete before the set threshold, then CloudWatch doesn't deliver your logs to your log group. You can set individual index level thresholds for each debug level (TRACE, DEBUG, INFO, and WARN).

For example, set the threshold for WARN debug level to 10 seconds for your index in OpenSearch Dashboards:

PUT /YOURINDEXNAME/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s"
}

Note: Replace YOURINDEXNAME with your index name.

For curl, run the following command:

curl -XPUT http://your_domain_endpoint/index/_settings -d '{"index.search.slowlog.threshold.query.level":"10s"}'

Note: Replace your_domain_endpoint with your domain endpoint. Replace 10s with the log threshold you want to use.

To log all the queries for your domain, you can set TRACE to 0 milliseconds. However, its resource intensive to log all queries and that might affect your domain performance. To turn off logging, set the threshold to -1.

To check your threshold, run the following command:

GET /YOURINDEXNAME/_settings?pretty

Note: Replace YOURINDEXNAME with your index name.

OpenSearch Service logs any queries that exceed the defined threshold.

Best practices

Take the following actions:

• Make configuration updates one at a time.
• To identify slow search queries, set a threshold for both the query phase and the fetch phase.
• Test with a low threshold value. Slowly increase the threshold to log only the queries that affect performance or require optimization.
• Choose the right number of shards for your cluster and use case to optimize cluster performance. For more information, see Shard strategy.
• For slow logs, turn on logging at the TRACE, DEBUG, INFO, and WARN debug levels. Each debug level logs different categories of information. Turn on logging according to the request status.

Apply slow log settings for all new indices at the cluster level

To apply slow logs settings for all new indices at the cluster level, use index templates. There isn't a cluster wide setting for index slow logs in OpenSearch Service.

Create an index template that includes the slow logs settings:

PUT _template/all_indices_template
{ 
  "index_patterns": ["*"], // This applies to all indices 
  "settings": { 
    "index.indexing.slowlog.threshold.index.warn": "10s",
    "index.indexing.slowlog.threshold.index.info": "5s",
    "index.indexing.slowlog.threshold.index.debug": "2s", 
    "index.indexing.slowlog.threshold.index.trace": "500ms" 
  }
}

The preceding template automatically applies the settings to all new indices that you create after the template is set up.

Related information

Analyzing Amazon Elasticsearch Service Slow Logs Using Amazon CloudWatch Logs Streaming and Kibana

How do I troubleshoot CloudWatch Logs so that it streams to my OpenSearch Service domain?

Viewing Amazon OpenSearch Service error logs