How do I resolve the "The security token included in the request is expired" error when I run Java applications on Amazon EC2?

Short description

AWS requires all application API requests to be digitally signed with credentials provided by AWS. When your application uses temporary credentials to create an AWS client, you must renew these credentials before they expire. If your credentials expire, then you receive the security token included in the request is expired error. To resolve this error, check your time reference, refresh expired temporary credentials, and then check your AWS Identity and Access Management (IAM) configurations.

Resolution

Make sure that your instance has a consistent time reference

Credentials expire if they use the incorrect time, so make sure that your server is accurate. Your EC2 instance must have a consistent and accurate time and date reference. Configure the Amazon Time Sync Service, or another Network Time Protocol (NTP) source on your instance. For more information, see Change the time zone of your instance.

Make sure that your temporary credentials aren't expired

If you use temporary credentials, then make sure that they're not expired. If the temporary credentials are expired, then you must generate a new set of temporary credentials and use those. It's a best practice to refresh temporary credentials five minutes before their expiration.

Check your IAM configuration

For applications that run on an EC2 instance, it's a best practice to use an IAM role assigned to the instance. If you use an IAM role, then check that your configuration is set up correctly.

With an IAM role, a default service constructor uses a default credentials provider chain to search for credentials in the following order:

1. In the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY system environment variables.
2. In the aws.accessKeyId and aws.secretKey Java system properties.
3. In the default credentials file.
4. In the instance profile credentials in the instance metadata associated with the IAM role.

For more information, see Use an IAM role to grant permissions to applications running on Amazon EC2 instances.

If you list credentials anywhere except the instance profile, then the default client constructor finds those credentials first. This configuration blocks the IAM role credentials. For more information, see Provide temporary credentials to the AWS SDK for Java.

To see the credentials for the IAM role, run the following commands from Windows PowerShell version 3.0 or later or from a Linux shell. If you use temporary credentials, then the following Windows and Linux commands show the latest temporary credentials for the instance.

Note: In the following commands, replace examplerole with the name of your IAM role.

Windows:

PS C:\> Invoke-RestMethod http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

You receive an output similar to the following example:

Code            : SuccessLastUpdated     : 2016-07-18T18:09:47Z
Type            : AWS-HMAC
AccessKeyId     : AKIAIOSFODNN7EXAMPLE
SecretAccessKey : wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Token           : token
Expiration      : 2016-04-27T22:39:16Z

Linux:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

You receive an output similar to the following example:

{    "Code" : "Success",
    "LastUpdated" : "2016-04-26T16:39:16Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
    "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "Token" : "token",
    "Expiration" : "2016-04-27T22:39:16Z"
}

If you receive a 404 error when you run the preceding curl command, then verify that HTTP proxy is off for the metadata IP address. Also, make sure that the instance profile is attached to the instance.

If the issue persists, then verify that the instance isn't making multiple concurrent requests or running multiple sessions in parallel. This scenario can cause the Instance Metadata Service (IMDS) to throttle the query. To mitigate this issue, use the retry with exponential backoff pattern.

To set up retries, modify AWS_METADATA_SERVICE_NUM_ATTEMPTS. To set options, use environment variables, the ~/.aws/config file, or the user's botocore session. For more information, see Configuration on the Boto3 documentation website.

Example:

AWS_METADATA_SERVICE_TIMEOUT = 10
AWS_METADATA_SERVICE_NUM_ATTEMPTS = 5

If you run the curl command in a docker container, then increase the http-put-response-hop-limit to 2. Run the following AWS Command Line Interface (AWS CLI) modify-instance-metadata-options command:

aws ec2 modify-instance-metadata-options --instance-id instance --http-put-response-hop-limit 2 --http-endpoint enabled

Note: Replace instance with your instance ID. If you receive errors when you run AWS CLI commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

For more information, see Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service.

Role credentials automatically rotate or refresh five minutes before the assigned temporary credentials expire.

Related information

Configure SDK authentication

IAM roles for Amazon EC2