How do I use NVMe over TCP to mount an FSx for ONTAP file system on an EC2 Linux instance?

Resolution

Note: The following resolution is for EC2 instances that run Red Hat Enterprise Linux (RHEL) 9.3.

Prerequisites:

• The instance allows outbound traffic on TCP port 4420.
• The NetApp ONTAP file system allows inbound traffic on TCP port 4420.
• Configure your FSx for ONTAP file system with NVMe/TCP.
  Note: For information about what you must do before you configure your file system with NVMe/TCP, see Before you begin.
• Use a second-generation file system that has six or fewer high-availability (HA) pairs.

Use NVMe/TCP to mount your file system on your RHEL instance

Complete the following steps:

1. To install the required packages, run the following command on the instance:

   sudo yum install -y nvme-cli

2. To load the NVMe/TCP kernel module, run the following command on the instance:

   sudo modprobe nvme-tcp

3. Retrieve the addresses of the block storage interfaces for the storage virtual machine (SVM) where you created your NVMe devices. Run the following command on the ONTAP server:

   network interface show -vserver svm_name -data-protocol nvme-tcp

   Note: Replace svm_name with your SVM name. Both iSCSI and NVMe/TCP use the iscsi_1 logical interchange format (LIF).
   Example output:

               Logical               Status         Network            Current                    Current Is 
   Vserver     Interface             Admin/Oper     Address/Mask       Node                       Port    Home
   ----------- ----------            ----------     ------------------ -------------              ------- ----
   svm_name
               iscsi_1               up/up          iscsi_1_IP/X       FSxId0123456789abcdef8-01  e0e     true
               iscsi_2               up/up          iscsi_2_IP/X       FSxId0123456789abcdef8-02  e0e     true
   2 entries were displayed.

4. To identify NVMe subsystems, run the following command on the instance:

   sudo nvme discover -t tcp -w client_IP -a iscsi_1_IP

   Note: Replace iscsi_1_IP with your iscsi_1 IP address and client_IP with the client IP address.

5. To connect to the NVMe subsystem, run the following command on the instance:

   sudo nvme connect-all -t tcp -w client_IP -a iscsi_1_IP -l 1800

   Note: Replace iscsi_1_IP with your iscsi_1 IP address and client_IP with the client IP address.

6. To verify that the connection works as expected, run the following command on the instance:

   sudo nvme list

   In the output, check the Model column for /dev/nvmeXny.

7. (Optional) To create a file system, run the following command on the instance:

   sudo mkfs.xfs /dev/nvmeXnY

   Note: To find /dev/nvmeXnY, check the output of the nvme list command. Find the Model column that has the NetApp ONTAP Controller value.

8. To create and mount to a mount point, run the following commands on the instance:

   sudo mkdir /mnt/fsx-nvme
   sudo mount /dev/nvmeXnY /mnt/fsx-nvme

   Note: Replace /mtn/fsx-nvme with your directory path.

9. (Optional) To configure automatic mount during boot, add the following line to the /etc/fstab file:

   /dev/nvmeXnY /mnt/fsx-nvme xfs _netdev,x-systemd.automount,x-systemd.requires=network-online.target 0 0

   Note: Replace nvmeXnY with your NVMe name and /mnt/fsx-nvme with your file path.
   Then, create a systemd service for NVMe connection with the /etc/systemd/system/ file path.
   Example file:

   [Unit]
   Description=NVMe connection to FSx
   After=network.target
   
   
   [Service]
   Type=oneshot
   RemainAfterExit=yes
   ExecStart=/usr/sbin/nvme connect -t tcp -n "nqn.1992-08.com.netapp:sn.abcde" -w port -a fs-ABCDE.fsx.region.amazonaws.com -s 4420
   ExecStop=/usr/sbin/nvme disconnect -n "nqn.1992-08.com.netapp:sn.abcde"
   
   
   [Install]
   WantedBy=multi-user.target

10. To activate the service, run the following command:

    sudo systemctl enable nvme-fsx.service

Troubleshoot NVMe/TCP mount issues

Important: It's a best practice to create an account on NetApp. You must have a NetApp account to access NetApp troubleshooting documentation.

If you experience issues when you mount your file system, then take the following troubleshooting actions:

• To check connectivity on the required port, run the following command:

  telnet fs-ABCDE.fsx.region.amazonaws.com 4420

  Note: Replace fs-ABCDE with your file system name, and region with your AWS Region.

• To check whether NVMe discovery works as expected, run the following command:

  sudo nvme discover -t tcp -w port -a fs-ABCDE.fsx.region.amazonaws.com -s 4420

  Note: Replace port with your port, fs-ABCDE with your file system name and region with your Region.

• To check for issues in your NVMe connections, run the following command:

  sudo nvme list

• To check the logs for errors, run the following command:

  dmesg | grep nvme or journalctl -u nvme-fsx

Configure the FSx for ONTAP server

To verify that the NVMe service is running, run the following commands:

vserver nvme show
vserver nvme subsystem show

To confirm that you correctly configured your namespace, run the following command:

vserver nvme namespace show

To verify that you added the host NVMe qualified name (NQN) to the NVMe subsystem, run the following command:

vserver nvme subsystem host show

To verify that the mapping between the namespace and the subsystem is correct, run the following command:

vserver nvme subsystem map show

Related information

How do I use NFS to mount an FSX for ONTAP volume on my EC2 Linux instance?

How do I mount an FSx for ONTAP CIFS share on a Linux EC2 instance that's not joined to an Active Directory?

How do I use iSCSI to mount an FSx for ONTAP iSCSI LUN on an EC2 Linux instance?