Invalid security token in Python container in ECS

Question

I am trying to access a DynamoDB table in one of my ECS containers via Python code using boto3. I keep getting an error in my container saying the Security token is invalid. However I have checked the following:
1. My ECS Task Execution Role has DynamoDB access
2. My ECS Task Role has DynamoDB access
3. This my code:
session = boto3.session.Session()
dynamo=session.resource('dynamodb')
table = dynamo.Table(TABLE_NAME)

If anyone could help me out with this issue, I would greatly appreciate it. We have deployed it using Fargate and ensured that both the roles are attached to the task definition.

Leeroy Hannigan · Answer

I believe your issue is caused by having stale credentials on the host, either in the form of exported environment variables or through configuring the CLI credentials through `aws configure`.

Credentials follow a chain, and it will pick up both environment variables and aws CLI credentials before using a role attached to your instance. To fix, make sure you do not have anything blocking the chain from picking up the execution role. You can delete any credentials at `.aws/credentials` and `unset` any environment variables which may hold `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

### Provider Chain

Credential provider | Description
---|---
AWS access keys 	| AWS access keys for an IAM user (such as AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY).
Federate with web identity or OpenID Connect - Assume role credential provider 	|Sign in using a well-known external identity provider (IdP), such as Login with Amazon, Facebook, Google, or any other OpenID Connect (OIDC)-compatible IdP. Assume the permissions of an IAM role using a web identity token from AWS Security Token Service (AWS STS).
IAM Identity Center credential provider 	|Get credentials from AWS IAM Identity Center.
Assume role credential provider 	|Get access to other resources by assuming the permissions of an IAM role. (Retrieve and then use temporary credentials for a role).
Container credential provider 	|Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) credentials. The container credential provider fetches credentials for the customer’s containerized application.
Process credential provider 	|Custom credential provider. Get your credentials from an external source or process, including IAM Roles Anywhere.
IMDS credential provider 	|Amazon Elastic Compute Cloud (Amazon EC2) instance profile credentials. Associate an IAM role with each of your EC2 instances. Temporary credentials for that role are made available to code running in the instance. The credentials are delivered through the Amazon EC2 metadata service.

Awaneendra Tiwari · Answer

You have already checked several important points. Even though you've set up the roles correctly, there might be a problem with how the credentials are being accessed or used within your container. If solution outlined in the comment section by "Leeroy Hannigan" doesn't work then here are a few things to check and try:

1. Verify AWS region configuration in the container.
2. Try explicitly using EC2 instance metadata for credentials.
3. Ensure IAM roles for tasks are enabled in your ECS cluster.
4. Double-check execution and task role ARNs in the task definition.
5. Look for any restrictive permissions boundaries on IAM roles.
6. If using a private subnet, confirm DynamoDB VPC endpoint is set up.
7. Enable DEBUG logging for boto3 for more detailed credential info.

Hope this helps.

Credential provider	Description
AWS access keys	AWS access keys for an IAM user (such as AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY).
Federate with web identity or OpenID Connect - Assume role credential provider	Sign in using a well-known external identity provider (IdP), such as Login with Amazon, Facebook, Google, or any other OpenID Connect (OIDC)-compatible IdP. Assume the permissions of an IAM role using a web identity token from AWS Security Token Service (AWS STS).
IAM Identity Center credential provider	Get credentials from AWS IAM Identity Center.
Assume role credential provider	Get access to other resources by assuming the permissions of an IAM role. (Retrieve and then use temporary credentials for a role).
Container credential provider	Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) credentials. The container credential provider fetches credentials for the customer’s containerized application.
Process credential provider	Custom credential provider. Get your credentials from an external source or process, including IAM Roles Anywhere.
IMDS credential provider	Amazon Elastic Compute Cloud (Amazon EC2) instance profile credentials. Associate an IAM role with each of your EC2 instances. Temporary credentials for that role are made available to code running in the instance. The credentials are delivered through the Amazon EC2 metadata service.

Invalid security token in Python container in ECS

Provider Chain

Relevant content