Our cluster was upgraded to version 1.25 and I installed the "Amazon EBS CSI Driver". In this case via terraform registry.terraform.io/terraform-aws-modules/eks/aws:
cluster_addons = {
coredns = {
most_recent = true
}
kube-proxy = {
most_recent = true
}
vpc-cni = {
most_recent = true
}
aws-ebs-csi-driver = {
most_recent = true
}
}
I do also see the addon being listed in the AWS console.
The problem is this:
failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-8db0fea3-7c23-45c2-8110-cf8f96e86d26": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: I4PhC3ZdhwQHCG7l6hlMIfCQVy6uWfVgBelIceHqpOjJTZvagN6w-kh0aUa55_D0OVny8x9ndNsTzbxeB5CK5Ys84pMfomwgdSygufMujbxHZSWNWIRyggLcOqy064kLGFeVp_HVOFyVy_pFCzCYRpCE-WBe5DsJEZ5ikkJwHKvvfU4KPRI8CMS8p_7he_84kOe-V0HJVLpglSNIcpAZFO42J5VC6BQgAnKxD-cLCTlvQ3B7jHi3kVpBli3ki_dkcD4xK2-O48cy9E32bNXN3bx5PTGMZ6JiCNQ128he5KM-SdkH3nBhgLUtVNp6QpiA_OIf7kuGGJBf_LxI4rxWqtnv11LsrIwj9rT1IszKShaxtzzM4oni2tP0ACg0hyZWpEKYXLtUy-5aDV6MuMORgGEA9n6hMJw4ZavSQEpV0IWnOwmdwIVORdVNaymtES6Gn5DwFmCw9IHYRpBoQuGfcrhtAdJoIp18ym-T5A-aqLHWCS140mMZWP6tB03SpsaIHeI7RCVOxgpFTMGla1aaRJQpqlIf0h-Ak4R4_5DfYn-Ep6HSJyx1DrBiCvkdJIGZIp0bJusgU9Ea6TP-iUH5pkZPAsSD5VihCX1MMCarw7zizJKGgKvtDqL3wOS9MPmquLwe4twhZ6UdBA status code: 403, request id: e268aa26-9697-4f27-832e-cb845bd59fca
I have found this page: https://repost.aws/knowledge-center/eks-troubleshoot-ebs-volume-mounts and indeed I don not have the "annotation" listed in my service account.
x@x:~/aws-ebs-csi-driver/examples/kubernetes/dynamic-provisioning$ kubectl describe sa ebs-csi-controller-sa -n kube-system
Name: ebs-csi-controller-sa
Namespace: kube-system
Labels: app.kubernetes.io/component=csi-driver
app.kubernetes.io/managed-by=EKS
app.kubernetes.io/name=aws-ebs-csi-driver
app.kubernetes.io/version=1.19.0
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
However, I never added the sa, it was added when the addon was installed. I tried removing / reinstalling it but no annotation. Am I missing a step or is this somehow broken?
Yes is is related (left a comment there, thanks) but it does not fix my issue. This reddit posts does https://www.reddit.com/r/Terraform/comments/znomk4/ebs_csi_driver_entirely_from_terraform_on_aws_eks/ but I'm not sure if that is the preferred way.