Why is my Amazon EBS volume stuck in the attaching state?

9 minute read

I attached my Amazon Elastic Block Store (Amazon EBS) volume to my Amazon Elastic Compute Cloud (Amazon EC2) instance. The Amazon EBS volume is still in the attaching state after 10-15 minutes.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To troubleshoot an Amazon EBS volume stuck in the attaching state, complete the following steps.

Verify the volume attachment limit for the instance

The maximum number of Amazon EBS volumes that your Amazon EC2 instance can have depends on the operating system (OS) and instance type. If your instance reaches the instance level limit, then change the instance type. There are instance volume limits for Linux and Windows instances. For more information, see Amazon EBS volume limits for Amazon EC2 instances.

To see how many volumes are attached to your instances, run the following describe-instances command:

aws ec2 describe-instances --instance-ids i-1234567890abcdef0 --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId'

To resolve attaching state issues, see Change the instance type for your Amazon EC2 instance.

Verify the device name for the Amazon EBS volume

When you attach an Amazon EBS volume to an EC2 instance, you can specify a device name for the volume. By default, the block device driver creates a name for you. The instance's block device driver assigns a name to the volume when you mount the volume. The assigned volume name can be different from the name that you create.

To verify the device name, complete the following steps in the Amazon EC2 console or AWS CLI:

Amazon EC2 console

Open the Amazon EC2 console, and then select your EC2 instance.
Under the Storage tab, look for the device name, such as /dev/sdf or /dev/xvda, and Volume ID that's listed for your attached Amazon EBS volume.
Verify the device name.

AWS CLI

To find the device name for volumes attached to your instance, run the describe-volumes command:

aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=YOUR_INSTANCE_ID --region YOUR_REGION

Note: Replace YOUR_INSTANCE_ID with your instance ID and YOUR_REGION with your AWS Region.

Connect to the instance, and then run the following command to see all block devices and their mount points:
```
lsblk
```

For more information, see Device names for volumes on Amazon EC2 instances.

Check that the device name for your Amazon EBS volume isn't already in use

When you attach an Amazon EBS volume, you specify a device name. For example, you could specify /dev/sdf. The instance's block device driver mounts the volume and could assign a different name internally. If the device name that you specified is already in use, then the attach operation fails. The volume remains in the attaching state.

This can happen for the following reasons:

The block device driver remaps the device name. On an EC2 instance launched from a Hardware Virtual Machine (HVM)-based Amazon Machine Image (AMI), the block device driver remaps /dev/sda1 to /dev/xvda. If you attach a secondary Amazon EBS volume to /dev/xvda, then the operation fails because the root volume already uses that device.
The block device driver hasn't released a device name. If you already used a device name to attach a volume, then the block device driver might still hold a reference to that name. In that case, you can't use the same name for the new volume.

To rule out that your device name isn't already in use, take the following actions based on your environment:

Linux

To check the device name that's in use by the instance, run the following command on the instance:

lsblk

Example output:

NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
  xvda    202:0    0   8G  0 disk
  └─xvda1 202:1    0   8G  0 part /
  xvdf    202:80   0  10G  0 disk /data
  xvdg    202:96   0  20G  0 disk

The example output shows that the instance uses /dev/xvda, /dev/xvdf, and /dev/xvdg. The attach operation fails if you use any of these device names to attach a new volume. Choose an unused device name, such as /dev/xvdh.

Nitro-based instances (NVMe)

Amazon EBS volumes appear as NVMe devices, such as /dev/nvme0n1, instead of /dev/xvd*.

To identify what NVMe device corresponds to the device name that you specified at attach time, run the following command:

lsblk -o NAME,SIZE,MOUNTPOINT,SERIAL

Example output:

NAME           SIZE MOUNTPOINT SERIAL
nvme0n1          8G            vol0abc1234def567890
└─nvme0n1p1      8G /
nvme1n1         10G /data      vol0def4567abc890123
nvme2n1         20G            vol0789abc123def4560

The SERIAL column shows the volume ID. To check if the instance already uses the device name, match the volume ID in the SERIAL column against the volume ID of the volume that you want to attach.

Windows

To view the disk configuration on your Windows EC2 instance, run the following PowerShell command:

Get-Disk | Format-Table Number, FriendlyName, SerialNumber, Size, PartitionStyle

Example output:

Number FriendlyName          SerialNumber             Size PartitionStyle
----- ------------          ------------             ---- --------------
  0     Amazon Elastic Block  vol0abc1234def567890    30 GB GPT
  1     Amazon Elastic Block  vol0def4567abc890123    50 GB GPT

The disks listed in the output correspond to an attached Amazon EBS volume. The SerialNumber maps to the volume ID without the vol- prefix. If the device name that you want to use is mapped to one of these disks, then choose a different device name.

For more information about available and recommended device names, see Device names for volumes on Amazon EC2 instances.

Troubleshoot Amazon EKS or Kubernetes volume attachment issues

If you use Amazon Elastic Kubernetes Service (Amazon EKS) with the Amazon EBS Container Storage Interface (CSI) driver, then your volumes can get stuck in the attaching state. The volumes get stuck because of the stale VolumeAttachment objects. Kubernetes creates stale VolumeAttachment objects when the cluster replaces a node or a node becomes unreachable, but Kubernetes doesn't clean up the VolumeAttachment resource for the old node. Stale objects can also cause multi-attach errors when the new node tries to attach the same volume.

To troubleshoot volume attachment issues, complete the following steps:

To check for stale attachments, run the following command:

kubectl get volumeattachments

Look for VolumeAttachment objects that reference a node that no longer exists or is in a NotReady state.

To delete the stale volume attachment, run the following command:

kubectl delete volumeattachment YOUR_ATTACHMENT

Note: Replace YOUR_ATTACHMENT with your attachment name.

The Amazon EBS CSI driver creates a new attachment to the correct node.

Note: In rare cases with large clusters, node replacements can generate enough Amazon EC2 API calls to cause throttling. If you see ThrottlingException errors in the Amazon EBS CSI controller Pod logs, then see Request throttling for the Amazon EC2 API.

For more information, see How do I troubleshoot issues with my Amazon EBS volume mounts in Amazon EKS?

Check for encryption or AWS KMS key permission issues

If your volume is encrypted with an AWS Key Management Service (AWS KMS) customer managed key, then the attach operation can fail silently. Then, your volume is stuck in the attaching state. The operation fails when the AWS Identity and Access Management (IAM) principal that performs the attach operation doesn't have the required KMS permissions.

Make sure the IAM user or role has the following permissions on the KMS key:

kms:Decrypt
kms:CreateGrant
kms:GenerateDataKeyWithoutPlaintext
kms:ReEncryptFrom
kms:ReEncryptTo
kms:DescribeKey

Also, check that the KMS key policy allows the principal to use the key. If the key is in a different account, then both the key policy and the IAM policy must grant the principal access to the KMS key. For more information, see Why can't I use a custom AWS KMS key to create or attach an encrypted Amazon EBS volume?

Force detach and reattach the volume

Important: Before you begin, back up your data. For more information, see Best practices for Amazon EC2.

If the previous resolutions don't work, then complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, choose Volumes, and then select the volume that's experiencing issues.
Choose Actions, and then choose Force detach volume.
Use a different device name and attach the volume to the EC2 instance again. For example, in Linux instances, use /dev/sdg instead of /dev/sdf. In Windows instances, use /dev/xvdg instead of /dev/xvdf.
Note: The instance must be in the running state.

If the previous steps to troubleshoot the issue don't work or if you must use a device name that isn't working, then complete the following steps:

Reboot the instance.
Stop and start the instance to migrate the instance to new hardware.

If the issue persists after you start and stop the instance, then contact AWS Support.

Important: When you stop an instance, you lose all data stored on the instance store volumes. For more information, see Instance store temporary block storage for EC2 instances and Root volumes for your Amazon EC2 instances.

A volume can get stuck in the attaching state because of one of the following issues with the Amazon EBS service:

The volume was previously working and the issue started even though you didn't modify the configuration.
You force detached and reattached the volume, but the issue persists.
Multiple volumes in the same Availability Zone are affected.

To check for service-related events in your AWS Account, Region, or Availability Zone that could cause issues, use the AWS Health Dashboard. If no events are listed and the issue persists, then contact Support.