AWS EKS instance not found when attaching EBS volume

0

A few days ago attaching EBS volumes suddenly stopped working. My EKS cluster uses ebs.csi.aws.com addon with dynamic provisioning.

here is my storageClass config

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

and volumeClaimTemplate in my sts config

  volumeClaimTemplates:
  - metadata:
      name: log
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

after sts deployment a PVC, PV and VolumeAttachment are created, however the pod is stuck in ContainerCreating state with error AttachVolume.Attach failed for volume "pvc-xxx" : rpc error: code = NotFound desc = Instance "i-xxx" not found

I triple-checked, the volume is not attached to any other instance, and the instance exists.

One funny thing though - when I describe the created PV I see this

Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            ebs.csi.aws.com
    FSType:            ext4
    VolumeHandle:      vol-xxx
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=xxx-8081-ebs.csi.aws.com

the (unmasked) volumeHandle does not even exist.

Where might be the problem? As I said earlier, this issue popped up from day to day without changing the config

K8S version 1.24 EBS CSI Driver addon version v1.11.5-eksbuild.2 (upgrade nor downgrade didn't help)

Thanks

1개 답변
2

When you use EBS for persistent volumes, you need remebmer, that EBS is located in a single AZ, so only EC2 instance from the same AZ will be able to attach it. If a pod is rescheduled to a node in another AZ, it may fail with an error that it can not find/attach a persistent volume. Every node can have a label with its AZ, so you can use nodeSelector or Affinity to make pod be scheduled only in particular AZ

profile picture
전문가
답변함 일 년 전
profile picture
전문가
Artem
검토됨 한 달 전
  • I believe the volume binding mode WaitForFirstConsumer should prevent AZ mismatch.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠