How do I troubleshoot my Amazon EBS volume that's stuck in the optimizing state after modification?

Short description

When you modify an Amazon EBS volume, it goes through a sequence of states: modifying, optimizing, and completed. The time to complete the sequence depends on the configuration change. Size increases take effect within seconds after the volume enters the optimizing state. Performance changes can take from a few minutes to several hours. A fully used 1-TiB volume takes about 6 hours to complete optimization.

Resolution

Note: You can't cancel a volume modification after you submit it. The volume remains fully usable during optimization, but performance is transitional between source and destination specifications.

Wait for the volume modification to complete

The volume enters the optimizing state shortly after modification begins, and the following occurs:

• Size increases are immediately available for use.
• Performance changes are in progress.
• The volume remains fully functional.

You can extend the file system without waiting for completion.

Modification completion times vary based on the following factors:

• A size-only change enters the optimizing state within a few seconds. Full optimization then takes 6–24 hours to complete.
• Performance changes take a few minutes to several hours to complete, depending on volume size and utilization.
• A fully used 1-TiB volume takes 6 hours to complete optimization.
• Volumes larger than 1 TiB take proportionally longer to optimize. For example, a fully used 2-TiB volume can take about 12 hours to optimize. A fully used 4-TiB volume can take 24 hours or more to optimize. The optimization time scales linearly with volume size.
• Uninitialized volumes, such as volumes that you created from snapshots and haven't fully read, can take more than 24 hours to complete optimization.

Distinguish between normal optimization and a stuck volume

Understand the difference between a volume that progresses normally through optimization and one that's stuck.

A volume that progresses normally through optimization shows the following characteristics:

• The progress percentage gradually increases over time, even if it increases slowly.
• The total optimization duration is within the expected range based on volume size. Expect optimization to complete in about 6 hours per TiB for fully used volumes.
• You can fully use the volume with transitional performance between source and destination specifications.
• Amazon CloudWatch I/O metrics such as VolumeReadOps and VolumeWriteOps show normal activity.

A volume that's potentially stuck shows the following characteristics:

• The progress percentage shows no change over several hours. For example, the value remains the same for 2–4 hours or more.
• The total duration significantly exceeds the expected time for the volume size. For example, more than 24 hours for a 1-TiB volume, or more than 48 hours for a 2-TiB volume.
• The volume is still usable, but performance doesn't reflect the new destination configuration even after an extended period.
• CloudWatch metrics show unusual I/O patterns, no I/O activity, or unexpected latency spikes.

To determine if your volume is stuck, monitor the modification progress percentage over 2–4 hours. To check progress, see Monitor the progress of Amazon EBS volume modifications. Also, check if you created the volume from a snapshot and haven't fully initialized it, because this condition can significantly extend optimization time.

Monitor the progress of your volume modification

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use the Amazon Elastic Compute Cloud (Amazon EC2) console or the AWS CLI to monitor the progress of your volume modification

Amazon EC2 console

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Volumes.
3. Select the volume.
4. On the Details tab, check the modification progress of the volume in the Volume state column.

AWS CLI

To view the progress of one or more volume modifications, run the following describe-volumes-modifications AWS CLI command:

aws ec2 describe-volumes-modifications --volume-ids vol-1234567890abcdef0

Example output:

{
    "VolumesModifications": [
        {
            "TargetSize": 200,
            "TargetVolumeType": "io1",
            "ModificationState": "optimizing",
            "VolumeId": "vol-1234567890abcdef0",
            "TargetIops": 10000,
            "StartTime": "2024-01-19T22:21:02.959Z",
            "Progress": 70,
            "OriginalVolumeType": "gp2",
            "OriginalIops": 300,
            "OriginalSize": 100
        }
    ]
}

To filter for volumes in the optimizing or completed state, run the following command:

aws ec2 describe-volumes-modifications \
    --filters Name=modification-state,Values="optimizing","completed" \
    --query "VolumesModifications[?StartTime>='2024-01-01'].{ID:VolumeId,STATE:ModificationState}"

For more information, see Monitor the status of Amazon EBS volume initialization.

Create a new volume if immediate modification is required

Important: The following resolution requires downtime for your Amazon EC2 instance.

If you must modify the volume when it's still in the optimizing state, then complete the following steps:

1. Create a new Amazon EBS volume from an existing snapshot or a new snapshot.
   Important: You must fully initialize the new Amazon EBS volume to achieve full volume performance. Empty volumes deliver maximum performance immediately, but you must initialize volumes created from snapshots.
2. Detach your current Amazon EBS volume from the instance.
3. Attach your new Amazon EBS volume to the instance.
4. Start the new instance, and then verify that the new volume works correctly.

Understand volume initialization for snapshot-based volumes

Note: If you created your volume from a snapshot before modifying it, then you might experience both optimization and initialization simultaneously.

Volume optimization and initialization are different processes.

Volume optimization occurs after you modify an existing Amazon EBS volume and has the following characteristics:

• Occurs after you modify volume size, type, IOPS, or throughput
• Shows a status of optimizing in the Amazon EC2 console
• Completes within 6–24 hours depending on size, but volumes larger than 1 TiB take longer
• Allows full volume use during the optimization process

Volume initialization occurs when you create a new Amazon EBS volume from a snapshot. This process has the following characteristics:

• Applies only to volumes that you create from Amazon EBS snapshots
• Shows a volume status of available, but loads blocks from the snapshot on first access
• Causes high I/O latency on the first read of each block
• Completes faster when you use Amazon EBS Provisioned Rate for Volume Initialization or fast snapshot restore (FSR)

Optimize volume initialization for faster performance

When you create a volume from a snapshot, you must initialize the volume before it can achieve maximum performance. To avoid initialization delays, use one of the following approaches.

Use Amazon EBS Provisioned Rate for Volume Initialization

Specify a volume initialization rate of 100–300 MiB/s when you create volumes from snapshots. This controls and accelerates the data transfer from Amazon Simple Storage Service (Amazon S3) to your Amazon EBS volume. Your volumes reach full performance within a predictable time frame.

Use a snapshot that's configured for FSR

Set up FSR so that your snapshots can create fully initialized volumes instantly with maximum provisioned performance from the moment of creation. FSR removes the initialization period and I/O latency penalties.

Manually initialize volumes

If you didn't activate a provisioned rate or FSR, then manually initialize volumes by reading all blocks before use. For example, use the dd or fio command.

Monitor volume initialization

Use the Amazon EC2 console or AWS CLI to track the initialization progress of your volumes to understand when they reach full performance.

Amazon EC2 console

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Volumes.
3. Select the volume that you want to check for initialization status.
4. The Initialization state field in the grid and Details tab displays progress information. For example, this field shows Initializing followed by the progress percentage, such as 75%.

AWS CLI

To view the initialization status, run the following describe-volume-status AWS CLI command:

aws ec2 describe-volume-status --volume-ids vol-1234567890abcdef0

Example output:

{
    "VolumeStatuses": [
        {
            "Actions": [],
            "AvailabilityZone": "us-east-1a",
            "Events": [],
            "VolumeID": "vol-1234567890abcdef0",
            "VolumeStatus": {
                "Details": [
                    {
                        "Name": "io-enabled",
                        "Status": "passed"
                    },
                    {
                        "Name": "io-performance",
                        "Status": "not-applicable"
                    },
                    {
                        "Name": "initialization-state",
                        "Status": "completed"
                    }
                ],
                "Status": "ok"
            },
            "InitializationStatusDetails": {
                "InitializationType": "provisioned-rate",
                "Progress": 75,
                "EstimatedTimeToCompleteInSeconds": 850
            }
        }
    ]
}

For more information, see Monitor the status of Amazon EBS volume initialization.

Contact Support

Contact AWS Support if you experience the following situations:

• Your volume is 1 TiB or smaller, has been in the optimizing state for more than 24 hours, and the progress percentage hasn't changed.
• Your volume is larger than 1 TiB and has been in the optimizing state for more than 48 hours. For larger volumes, allow about 6 hours per TiB. The optimization exceeds twice the expected duration with no progress change.
• Your volume experiences performance degradation that falls below the source volume specifications during the transitional period. If you downgrade IOPS, then performance shouldn't fall below the destination volume specifications.
• The modification state shows failed instead of optimizing or completed even after you retried the volume modification.
• The modification progress shows 0% for more than 6 hours after you initiated the modification.

Related information

Amazon EBS now supports up to four Elastic Volumes modifications in 24 hours

Why isn't my EBS volume increase reflected in my OS or Disk Management?