ALB load balancer controller start up errors


When the ALB controller pods come up i see below errors. looks like target group binding was deleted abnormally and i am stuck with it. How to get ride of this?

{"level":"error","ts":"2024-02-05T21:44:29Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"ngd-dashboard-tgb","namespace":"ingress-nginx"},"namespace":"ingress-nginx","name":"ngd-dashboard-tgb","reconcileID":"5841bf6b-3d92-471c-bd22-2d9c917963b8","error":"WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 7258cf29-b0ac-46e4-a186-135d85996a26"}
asked 3 months ago227 views
1 Answer

The error indicates an issue with the trust relationship between the IAM role used by the AWS Load Balancer Controller pod and the OIDC provider for your EKS cluster. A few things to check:

Verify the IAM role trusts the correct OIDC provider for your EKS cluster and allows the sts:AssumeRoleWithWebIdentity action. It should reference the pod's service account ARN and audience.

Confirm the OIDC provider ID matches what was used when creating the EKS cluster. You can get this from the EKS console.

Check the AWS Load Balancer Controller deployment is using the correct service account name and namespace.

Make sure any mutating webhooks are not modifying the pod's service account details.

To troubleshoot further, you can describe the IAM role trust policy and also get the mutating webhook configurations:

aws iam get-role --role-name ROLE_NAME kubectl get mutatingwebhookconfiguration

profile picture
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions