Why can't I use an IAM role for the service account in my Amazon EKS Pod?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Verify that you have an IAM OIDC identity provider for your Amazon EKS cluster

Create an IAM OpenID Connect (OIDC) provider for your cluster, if you don't already have one. You must have an OIDC provider for your cluster to use an IRSA.

Then, complete the following steps to verify that you correctly configured the OIDC provider:

1. Open the IAM console.
2. From the navigation pane, choose Identity providers.
3. Under Provider, identify and note the OIDC provider URL.
4. In a separate tab or window, open the Amazon EKS console.
5. From the navigation pane, choose Clusters.
6. Select your cluster, and then choose the Configuration tab.
7. Under Details, check the value of OpenID Connect provider URL. Verify that it matches the OIDC provider URL in the IAM console.
8. If the URLs don't match, then you must create a new IAM OIDC provider.

Validate your IAM role policies and trust policy configuration

IAM issues occur when your IAM role doesn't have all the required permissions. Or, your IAM role's trust relationship policy might have syntax errors if you created your IAM role with the AWS Management Console or AWS CLI.

To validate your IAM role policies and check for syntax errors in your trust policy, complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Roles, and then select your role.
3. Choose the Permissions tab, and then verify that you assigned all permissions required for your configuration to the role.
4. Choose the Trust relationships tab, and then choose Edit trust relationship.
5. In the policy document for your trust relationship, verify that the format of your policy matches the format of the following JSON policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::your-account-id:oidc-provider/oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:sub": "system:serviceaccount:your-namespace:your-service-account",
             "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:aud": "sts.amazonaws.com"
           }
         }
       }
     ]
   }

   In your JSON policy, check the format of the Federated property line and the StringEquals property line. For Federated, confirm that you correctly formatted the AWS Region code, AWS account ID, and unique OIDC identifier. For StringEquals, confirm that you correctly formatted the Region code, OIDC unique identifier, Kubernetes namespace, and Kubernetes service account name.
6. If you edit your policy document to correct formatting errors, then choose Update trust policy.

Confirm that your service account exists and has a correctly formatted annotation for the IAM role's Amazon Resource Name (ARN)

Complete the following steps:

1. To confirm that your Kubernetes service account exists, run the following command:

   kubectl get serviceaccount YOUR_ACCOUNT_NAME -n YOUR_NAMESPACE -o yaml

   Note: Replace YOUR_ACCOUNT_NAME with your account name, and YOUR_NAMESPACE with your namespace.
   If the preceding command doesn't return a service account name, then create a service account. For more information, see Use more than one ServiceAccount on the Kubernetes website.
2. Confirm that your service account uses the name that you expect. Also, confirm that you correctly formatted its role-arn annotation. Example:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/my-example-iam-role
     name: my-example-serviceaccount
     namespace: my-test-namespace

Use a test Pod to verify that the service account works

Run a test Pod to verify that the service account works correctly, can mount environment variables, and can assume the specified IAM role. Complete the following steps:

1. Create a local YAML file called awscli-pod.yaml. Example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: awscli
     labels:
       app: awscli
   spec:
     serviceAccountName: YOUR_SERVICE_ACCOUNT
     containers:
     - image: amazon/aws-cli
       command:
         - "sleep"
         - "604800"
       imagePullPolicy: IfNotPresent
       name: awscli
     restartPolicy: Always

   Note: Replace YOUR_SERVICE_ACCOUNT with your Kubernetes service account name.

2. To create the test Pod from the YAML file in your namespace, run the following command:

   kubectl apply -f ./awscli-pod.yaml -n YOUR_NAMESPACE

   Note: Replace YOUR_NAMESPACE with your namespace.

3. To confirm that the awscli pod has the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables, run the following command:

   kubectl exec -n YOUR_NAMESPACE awscli -- env | grep AWS

   Example output:

   AWS_ROLE_ARN=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME
   AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

4. To confirm that the test Pod uses the correct IAM role, run the following command:

   kubectl exec -it awscli -n YOUR_NAMESPACE -- aws sts get-caller-identity

   Example output:

   {    "UserId": "REDACTEDY471234567890:botocore-session-1632772568",
       "Account": "012345678912",
       "Arn": "arn:aws:sts::012345678912:assumed-role/your-iam-role/botocore-session-1632772568"
   }

   Note the value for Arn. Make sure that the IAM role is the role that you want to use in the Pod.

5. To delete the awscli pod, run the following command:

   kubectl delete -f ./awscli-pod.yaml -n YOUR_NAMESPACE

   Note: Replace YOUR_NAMESPACE with your namespace.

If the awscli Pod shows the correct IAM role, then the IAM roles for service accounts feature is working as expected. The preceding steps confirm that you correctly mounted the IAM token to the Pod. If your application still can't use the IAM token file, then there might be an issue at the application or SDK level. For example, there might be issues with how the application uses AWS credentials or with unsupported SDK versions. For more information, see Using the default credential provider chain and Use IRSA with the AWS SDK.