Why did my Amazon DocumentDB instance restart, recover, or failover?

8 minute read

Content level: Intermediate

I want to know the root cause for the restart, recover, or failover of my Amazon DocumentDB (with MongoDB compatibility) instance.

Short description

The Amazon DocumentDB instance automatically performs a restart, recovery, or failover under the following conditions:

Memory exhaustion occurred on the instance.
A replica fell behind the writer and restarted to catch up.
A scheduled engine patch inside maintenance window restarted the instances.
The DB instance class type was changed as part of a vertical instance scaling activity.
The user requested a failover of the DB instance.
The primary host of the Amazon DocumentDB instance is unreachable due to loss of network connectivity.

When the instance shows potential issues and fails to respond to Amazon DocumentDB health checks, Amazon DocumentDB automatically takes one of the following actions:

For a cluster with replicas, Amazon DocumentDB initiates a failover by promoting a replica to the new primary.
For a single-instance cluster with no replicas, Amazon DocumentDB recovers and replaces the instance in place on the same cluster storage volume.

Then, Amazon DocumentDB restarts the instance so that you can resume database operations as quickly as possible without administrative intervention.

Resolution

To identify the root cause of a downtime, you can view the Amazon DocumentDB events for the last 24 hours in the console. To retrieve events from the past 14 days, use the AWS CLI or API. For more information, see Managing Amazon DocumentDB events. When your instance restarts, recovers, or fails over, you see one of the following messages in your Amazon DocumentDB event notifications.

The instance ran out of memory

This event message indicates that the operating system terminated the database process due to memory exhaustion, which caused the instance to restart or failover. In the Amazon DocumentDB events, you see the following message:

The database process was killed by the OS due to excessive memory consumption. It is recommended to tune your database workload and/or parameter usage to reduce memory consumption.

To check if memory exhaustion caused the restart, review the following CloudWatch metrics around the restart timestamp: FreeableMemory (a sustained drop to low values indicates memory exhaustion), SwapUsage (a rise that mirrors the FreeableMemory drop confirms memory pressure) and DatabaseConnections (a spike in connections before the restart indicates memory was consumed by session allocations; each connection reserves a fixed amount of memory for session state).

Additionally when an instance experiences high memory pressure, Amazon DocumentDB throttles incoming requests to safeguard resources for critical processes such as health checks. Some operations are queued until the memory pressure subsides, and if memory pressure continues, queued operations time out.

Monitor for low-memory throttling with the following CloudWatch metrics:

LowMemNumOperationsThrottled
LowMemNumOperationsTimedOut
LowMemThrottleQueueDepth
LowMemThrottleMaxQueueDepth

If you see sustained throttling, scale up the instance to provide additional memory for your workload.

A replica fell behind the writer and restarted to catch up

This event message indicates that a replica could not keep pace with the writer's write activity. The writer stays up and the affected reader restarts. In the Amazon DocumentDB events, you see one of the following messages:

Read replica has fallen behind the master too much. Restarting database.
Read replica has been disconnected from master. Restarting database.

To check if this is the cause of restart, review the following CloudWatch metrics: DBInstanceReplicaLag (an increasing value before the restart confirms the reader was falling behind), and CPUUtilization and DatabaseConnections on the writer (high values indicate writer overload that the reader could not keep up with).

Applying modification to database instance class

This event message indicates that an instance class modification was applied as part of a vertical instance scaling activity. When you modify the writer's instance class, Amazon DocumentDB performs a failover to reduce downtime via switching the writer role to a reader instance with the highest priority (the lowest numbered) failover tier. It then applies the class change on the original instance which restarts as a reader. You also see the following related events:

On the instance being modified:

Applying modification to database instance class.
Finished applying modification to DB instance class.
A new writer was promoted. Restarting database as a reader.

On the cluster:

Started cross AZ failover to DB instance: your-instance-name.
Completed failover to DB instance: your-instance-name.

The user requested a failover of the DB instance

This message indicates that a user used the console, CLI, or API to initiate a manual failover. In Amazon DocumentDB, rebooting an instance does not result in a failover. To test failover, use the failover-db-cluster CLI command. For more information, see Amazon DocumentDB failover. You see the following events on the cluster:

Started cross AZ failover to DB instance: your-instance-name.
Completed customer initiated failover to DB instance: your-instance-name.

The primary host of the Amazon DocumentDB instance is unhealthy

This reason indicates that a transient underlying hardware issue or a transient network issue led to the loss of communication to the primary instance. The Amazon DocumentDB monitoring system could not communicate with the instance to perform health checks, which triggered a failover or recovery.

In the Amazon DocumentDB events, you see the following messages on the cluster:

Started cross AZ failover to DB instance: your-instance-name.
Completed failover to DB instance: your-instance-name.

On the instance:

Recovery of the DB instance has started. Recovery time will vary with the amount of data to be recovered.
DB instance restarted.
Recovery of the DB instance is complete.

If no other event (such as OOM, replica lag, or instance class modification) precedes the restart, the cause is typically an underlying infrastructure or network issue. Check the AWS Health Dashboard for any reported events in the instance's Availability Zone that overlap the restart timestamp.

Database cluster engine version has been upgraded

This message indicates that the cluster underwent a patch during its maintenance window. Amazon DocumentDB notifies you approximately 30 days before the patch auto-applies via the AWS Health Dashboard. Engine patches apply at the cluster level using the cluster's maintenance window. For more information, see Maintaining Amazon DocumentDB.

In the Amazon DocumentDB events, you see the following messages on the cluster:

Database cluster engine patch version upgrade started. Cluster remains online.
Database cluster engine version upgrade finished on all instances.
Database cluster engine version has been upgraded.

Your cluster is offline for a brief period while the engine version is upgraded. To find pending maintenance actions, run the following AWS CLI command:

aws docdb describe-pending-maintenance-actions

Best practices when dealing with DocumentDB instance outages

Review the following best practices to reduce downtime and prevent recurrence for your Amazon DocumentDB cluster:

To reduce downtime during an outage, run at least one replica in a second Availability Zone. For more information, see Understanding Amazon DocumentDB cluster fault tolerance.
Adjust the cluster maintenance window. If DocumentDB needs to apply system changes, the cluster is unavailable during this time for only the minimum amount of time required to make the necessary changes. For more information, see Maintaining Amazon DocumentDB.
Allocate sufficient resources to your database to run queries. The number of resources allocated depends on the instance class. If the instance frequently restarts because of a lack of resources, scale up your instance class to keep up with your application demands. For more information, see Managing instance classes.
To avoid instance throttling, configure Amazon CloudWatch alarms on key DocumentDB metrics that indicate the availability and health status of your instances. For example, set a CloudWatch alarm on the FreeableMemory metric so that you receive a notification when available memory drops below a safe threshold.
To have Amazon DocumentDB notify you when there is a failover or restart, create an event subscription. For more information, see Using Amazon DocumentDB event subscriptions.
To optimize database performance, make sure that your queries use appropriate indexes. Otherwise, you might experience performance issues and extended wait times. For more information, see Profiling Amazon DocumentDB operations.
Connect to your cluster using the cluster endpoint in replica set mode to minimize the impact of a failover on your application. For more information, see Connecting to Amazon DocumentDB as a replica set.
Test failover for your cluster to understand how long the process takes for your use case. For more information, see Amazon DocumentDB failover.
To troubleshoot loads in terms of CPU or memory, see Resolve high CPU utilization in Amazon DocumentDB.
If no other event (such as OOM, replica lag, or instance class modification) precedes the restart, the cause is typically an underlying infrastructure or network issue. Check the AWS Health Dashboard for any reported events in the instance's Availability Zone. If no health event is listed and the cause remains unclear, contact AWS Support for further investigation.