EBS CSI Driver Can't create PV in Amzon linux 2023

Question

hi,
 i have a kubernetes cluster with  AL2 node. i am migrated from  AL2 to AL2023. and i used to create and attach PV in ebs csi driver EKS Add-on. after i migrate AL2023 that ebs driver can't create PV.  AL2023 use "IMDSv2=Required" default. when i changed "IMDSv2=Optional" in ec2 instance ebs driver create pv. 
please if anyone know about how to fix this without changing "IMDSv2=Required".

below are the ebc csi pods error logs

```
I0719 09:48:28.576191       1 controller.go:133] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0719 09:48:33.577709       1 driver.go:107] "GRPC error" err="rpc error: code = Internal desc = Could not create volume \"pvc-e6b24f8e-2d6f-44ee-9b7b-a77bda12a511\": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded"
I0719 09:48:41.580169       1 controller.go:133] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0719 09:48:46.581166       1 driver.go:107] "GRPC error" err="rpc error: code = Internal desc = Could not create volume \"pvc-e6b24f8e-2d6f-44ee-9b7b-a77bda12a511\": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded"
I0719 09:49:02.583738       1 controller.go:133] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0719 09:49:07.587723       1 driver.go:107] "GRPC error" err="rpc error: code = Internal desc = Could not create volume \"pvc-e6b24f8e-2d6f-44ee-9b7b-a77bda12a511\": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded"
I0719 09:49:39.590219       1 controller.go:133] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0719 09:49:44.591730       1 driver.go:107] "GRPC error" err="rpc error: code = Internal desc = Could not create volume \"pvc-e6b24f8e-2d6f-44ee-9b7b-a77bda12a511\": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded"

```

sunerpy · Answer

I believe this is caused by the hop limit restriction in IMDSv2. You can try adjusting the HttpPutResponseHopLimit to 2, and you can  use a launch template to start new cluster instances.

iBehr · Answer

Make sure the AL2023 instance has the IAM Profile set (same as the AL2 instance) so that it is able to get permissions to call EC2 APIs.

Hope this helps!

Sandeep · Answer

Hello,

The error logs indicate the EBS CSI driver is unable to create volumes due to a failed attempt to retrieve credentials from the Instance Metadata Service Version 2 (IMDSv2).

Verify IMDSv2 on AL2023 Nodes:

Run curl -v http://169.254.169.254/latest/meta-data/iam/security-credentials on an AL2023 node.
A successful response indicates IMDSv2 is functional.

Check EBS CSI Driver Compatibility:

Refer to the driver's documentation (search online for the specific driver version) to confirm compatibility with AL2023 and its reliance on IMDSv2.

Kubernetes Documentation (search for "EBS CSI Driver"): https://kubernetes.io/docs/home/

Amazon EC2 Instance Metadata Service Version 2: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

EBS CSI Driver Can't create PV in Amzon linux 2023

Relevant content