Can I Use NVMe Reservation on a Multi-Attach Enabled Volume?



I want to use Amazon EBS Multi-Attach to share data between multiple EC2 instances.

In the UserGuide, it says

Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency.

I've googled "I/O fencing" and found that NVMe Reservation is a good way (compared to "power fencing", which means powering off the error node) to implement I/O fencing. However, I failed to use NVMe Reservation on a Multi-Attach Enabled Volume.

The details are as follows.

  • EC2: r5b.large, ubuntu-22.04
  • EBS: io2, 100GiB, 6000 IOPS, enabled Multi-Attach

I run following command in ec2 instance:

> sudo nvme list

and get

Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1           8.59  GB /   8.59  GB    512   B +  0 B   2.0
/dev/nvme1n1          volxxxxxxxxxxxxxxxx Amazon Elastic Block Store               1         107.37  GB / 107.37  GB    512   B +  0 B   2.0

then run

> sudo nvme resv-register -n 1 --crkey=0x0 --nrkey=0xa1 --rrega=0 /dev/nvme1n1

and get

NVMe status: INVALID_OPCODE: The associated command opcode field is not valid(0x2001)

I googled this error message and find that it may be caused by the driver not supporting. To confirm that, I run

> sudo nvme amzn id-ctrl /dev/nvme1n1 -H

and get

oncs      : 0
  [8:8] : 0	Copy Not Supported
  [7:7] : 0	Verify Not Supported
  [6:6] : 0	Timestamp Not Supported
  [5:5] : 0	Reservations Not Supported
  [4:4] : 0	Save and Select Not Supported
  [3:3] : 0	Write Zeroes Not Supported
  [2:2] : 0	Data Set Management Not Supported
  [1:1] : 0	Write Uncorrectable Not Supported
  [0:0] : 0	Compare Not Supported

The fifth bit indicates that "not supporting reservation".

So does NOT AWS Multi-Attach Enabled Volume support NVMe Reservation? Or there are other ways to solve this problem?

asked 2 years ago642 views
3 Answers

Multi-Attach enabled io2 volumes support I/O fencing. Support for NVMe reservations is enabled by default for all Multi-Attach enabled io2 volumes created after September 18, 2023. To enable support for NVMe reservations for existing io2 volumes created before September 18, 2023, you must detach all instances from the volume and then reattach the required instances. All attachments made after detaching all of the instances will have NVMe reservations enabled. NVMe reservations is supported with the following operating systems:

SUSE Linux Enterprise 12 SP3 and later
RHEL 8.3 and later
Amazon Linux 2 and later
answered 8 months ago

Yes! EBS Multi-Attach volumes support NVMe reservations.

You can learn more about the feature, supported commands, and how to set up Windows clusters in the EBS user guide and Windows driver guide.

Andy B
answered 8 months ago
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions