How do I update a service using Kube2IAM to use IRSA instead?

Question

I have a service running in a namespace that has been using Kube2IAM in order to assume a role that gives it access to S3.

I am trying to remove Kube2IAM and replace it with IRSA to assume the same role through a service account.
I have an OIDC provider and I have updated the role as per the documentation:

ARN:` arn:aws:iam::{{account-id}}:role/{{namespace}}-{{serviceName}}`

Permission Policy: `AmazonS3FullAccess`

Trust Relationship:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::{{account-id}}:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/{{id}}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/{{id}}:sub": "system:serviceaccount:{{namespace}}:{{serviceName}},
                    "oidc.eks.us-east-1.amazonaws.com/id/{{id}}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
```

I have then updated the configuration for my service in kuberneties to use the service account: 
```
serviceAccount:
  enabled: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::{{account-id}}:role/{{namespace}}-{{serviceName}}
```

However when my service tries to upload to S3 I get:

`com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: JXJP50TFCCP7J90S; S3 Extended Request ID: H2SpLJ5yEoPcpee91y5mNyEozIChJ8cGZYnoRiabW7Bkzn7/FZDqxolsPnFCeE0mv6dvpgn+E6g=)`

The service itself is using the AWS JDK SDK v1 and I have confirmed that it is running at least the version 1.11.704 (as per: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html).
I have attempted to query CloudTrail to find additional information about why the request is resulting in a 403 but no query has yielded any results, not even when running the service using the kube2IAM method I was previously using. so at this point I cannot tell if the service is actually assuming the role or if there is some other issue.

Is there anything I might have missed in order to get this to work?

Answer

Hello,

I see you have existing Identity Provider and referenced it in Role Trust Relationship. also, you have annotated the Service Account.

When you assign this Service Account to your Deployment, your pod will have few environment variables added by Pod Identity Webhook in your Cluster.

Can you describe the pod and check for Environment variables "AWS_DEFAULT_REGION", "AWS_REGION", "AWS_ROLE_ARN", "AWS_WEB_IDENTITY_TOKEN_FILE"

```
kubectl get pod -o json | jq -r '.spec.containers | .[].env'
```

[1] https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/

How do I update a service using Kube2IAM to use IRSA instead?

Relevant content