Unable to mount PVC to the pod that leverages the EBS Storage class in Amazon EKS

0

Hello,

We've AWS EKS cluster running with Kubernetes version 1.24 and with Amazon EBS CSI Driver's version v1.19.0. Currently I'm trying to leverage AWS EBS storage as the PV using the instructions provide here in this link https://repost.aws/knowledge-center/eks-persistent-storage.

  • [Successful]
    • I'm able to create EBS storage class object resource with following
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
 type: gp2
 csi.storage.k8s.io/fstype: ext4
- I'm able to create PVC object as well 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
  namespace: ebs-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1Gi
  • [Unsuccessful/Issue]
  • Given PVC above, now I'm trying to use the storage from the pod but getting this error:
Normal   SuccessfulAttachVolume  3m14s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-******"
  Warning  FailedMount             73s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage kube-api-access-9qtds]: timed out waiting for the condition
  Warning  FailedMount             64s (x9 over 3m12s)  kubelet                  MountVolume.MountDevice failed for volume ""pvc-******" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/********/globalmount": format of disk "/dev/nvme1n1" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bb41e5ca7e3b6a43537cee9ad3461e53770a21b6ca86fde5c33d55319b185003/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.42.9 (28-Dec-2013)
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/nvme1n1 is mounted.
/dev/nvme1n1: Device or resource busy while setting up superblock
)

Any guidences would be appriciated here....

3 Answers
0

Hi. I am not sure whether manifests you are using are correct. So can you follow Test the Amazon EBS CSI driver Section of following link, and give me result? (You have to delete existing sc and pvc before do this) https://repost.aws/knowledge-center/eks-persistent-storage

profile picture
EXPERT
answered 10 months ago
  • Also FYI, this is the AMI that I'm using in my EKS Cluster Nodes, amazon-eks-node-1.24-v20230513

  • Below are the results:

    [dynamic-provisioning]$ kubectl apply -f manifests/ -n ebs-test
    persistentvolumeclaim/ebs-claim created
    pod/app created
    storageclass.storage.k8s.io/ebs-sc created
    
    [dynamic-provisioning]$ kubectl describe storageclass ebs-sc
    Name:            ebs-sc
    IsDefaultClass:  No
    Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"ebs-sc"},"provisioner":"ebs.csi.aws.com","volumeBindingMode":"WaitForFirstConsumer"}
    
    Provisioner:           ebs.csi.aws.com
    Parameters:            <none>
    AllowVolumeExpansion:  <unset>
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     WaitForFirstConsumer
    Events:                <none>
    
    [dynamic-provisioning]$ kubectl get pods --watch
    NAME   READY   STATUS              RESTARTS   AGE
    app    0/1     ContainerCreating   0          113s
    
  • Output of the pod status being described with kubectl describe po app

    Events:
      Type     Reason       Age                   From     Message
      ----     ------       ----                  ----     -------
      Warning  FailedMount  56m (x3 over 61m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[kube-api-access-462ls persistent-storage]: timed out waiting for the condition
      Warning  FailedMount  7m56s (x35 over 62m)  kubelet  MountVolume.MountDevice failed for volume "pvc-f69eb6e6-3e79-471e-8d66-60bb2224bdce" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/087ff72572f4a3cf848dfb7a9f9a363753c2e5b92772c280df42671729e6cb56/globalmount": format of disk "/dev/nvme1n1" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/087ff72572f4a3cf848dfb7a9f9a363753c2e5b92772c280df42671729e6cb56/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.42.9 (28-Dec-2013)
    ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/nvme1n1 is mounted.
    /dev/nvme1n1: Device or resource busy while setting up superblock
    )
      Warning  FailedMount  2m9s (x17 over 54m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage kube-api-access-462ls]: timed out waiting
    
  • Thanks.Hmm..So it doesn't depends on manifest.

    Have you tried this Tshoot guide? https://repost.aws/knowledge-center/eks-troubleshoot-ebs-volume-mounts

    FYI:I cant find any issues on github

    https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues

0

Hi, I created it while not using this line, my manifest below:

csi.storage.k8s.io/fstype: ext4
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com 
parameters:
  type: gp3
  encrypted: 'true' 
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
profile picture
answered 9 months ago
0

Update: Thank you, guys, for your inputs in trying to find the resolution here! Much appreciated it!

The issue was related to EKS clusters having iSCIS enabled. For those who's running into the similar issue, you can check the issues in details here: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1417

raj
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions