Why do I get the "WebIdentityErr" error when I use the AWS Load Balancer Controller in Amazon EKS?

Short description

When you use the AWS Load Balancer Controller in Amazon EKS, you might receive the following error:

"failed to find existing LoadBalancer due to WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403"

The "WebIdentityErr" error occurs for the following reasons:

• Incorrect service account configurations
• Incorrect trust relationship of the AWS Identity and Access Management (IAM) role that you use in the service account

When you use the AWS Load Balancer Controller, you must use IAM permissions to allow worker nodes to manage Application Load Balancer or Network Load Balancer resources. To set up the IAM permissions, use IAM roles for service accounts (IRSA), or attach the IAM permissions directly to the worker node's IAM roles. For more information, see AWS Load Balancer Controller installation on the Kubernetes website.

Resolution

Incorrect service account configurations

To check whether you correctly configured your service account, complete the following steps:

1. To verify the service account name that's defined in your deployment, run the following command:

   kubectl describe deploy aws-load-balancer-controller -n kube-system | grep -i "Service Account"

2. To check the service account's IAM role Annotation, run the following command:

   kubectl describe sa aws-load-balancer-controller -n kube-system | grep "role-arn"

   Example output:

   Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::abcdefgxyz:role/AMAZON_EKS_LOAD_BALANCER_CONTROLLER_ROLE

3. If the IAM role annotation is missing or incorrect, then run the following command to update the annotation:

   kubectl annotate serviceaccount -n SERVICE_ACCOUNT_NAMESPACE SERVICE_ACCOUNT_NAME eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME

   Note: Replace SERVICE_ACCOUNT_NAMESPACE with your namespace, SERVICE_ACCOUNT_NAME with your account name, ACCOUNT_ID with your AWS account ID, and IAM_ROLE_NAME with your IAM role name. Also, make sure that you correctly associated the IAM role to a service account.

4. To restart the AWS Load Balancer Controller deployment to refresh Pod credentials, run the following command:

   kubectl rollout restart deployment/aws-load-balancer-controller -n kube-system

Incorrect trust relationship between the IAM role and the service account

When you establish the trust relationship between your IAM role and service account, you might encounter issues. To troubleshoot these issues, take the following actions based on common mistakes that can occur when you establish the trust relationship.

You didn't correctly define the IAM role or trust relationship for the "sts:AssumeRoleWithWebIdentity" action

Verify that you defined the trust relationship for the sts:AssumeRoleWithWebIdentity action and not the sts:AssumeRole action.

The following example is a trust relationship that isn't correctly defined:

{      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "abcde.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
}

To resolve this issue, update the trust relationship for the sts:AssumeRoleWithWebIdentity action. Example policy:

{  "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC-PROVIDER_ID"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"
          }
        }
      }
    ]
  }

Note: Replace AWS_ACCOUNT with your account ID, REGION with your AWS Region, and EKS_CLUSTER_OIDC-PROVIDER_ID with your OpenID Connect (OIDC) ID. Also, replace LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT with the service account name that's defined in your AWS Load Balancer Controller deployment.

To use the same IAM role for multiple clusters in one account, use a trust relationship similar to the following example:

{  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::AWS-ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC-PROVIDER_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",
          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

You didn't add the OIDC provider ID when you created an Amazon EKS cluster

Verify that you created and associated the OIDC provider ID with your Amazon EKS cluster.

You incorrectly entered the service account name

To review your AWS Load Balancer Controller deployment, run the following command:

kubectl describe deploy aws-load-balancer-controller -n kube-system | grep -i "Service Account"

If needed, update your deployment to enter the correct service account name.