A few days ago attaching EBS volumes suddenly stopped working.
My EKS cluster uses ebs.csi.aws.com addon with dynamic provisioning.
here is my storageClass config
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
and volumeClaimTemplate in my sts config
volumeClaimTemplates:
- metadata:
name: log
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
after sts deployment a PVC, PV and VolumeAttachment are created, however the pod is stuck in ContainerCreating state with error AttachVolume.Attach failed for volume "pvc-xxx" : rpc error: code = NotFound desc = Instance "i-xxx" not found
I triple-checked, the volume is not attached to any other instance, and the instance exists.
One funny thing though - when I describe the created PV I see this
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: ebs.csi.aws.com
FSType: ext4
VolumeHandle: vol-xxx
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=xxx-8081-ebs.csi.aws.com
the (unmasked) volumeHandle does not even exist.
Where might be the problem? As I said earlier, this issue popped up from day to day without changing the config
K8S version 1.24
EBS CSI Driver addon version v1.11.5-eksbuild.2 (upgrade nor downgrade didn't help)
Thanks
I believe the volume binding mode WaitForFirstConsumer should prevent AZ mismatch.