Automation: AWS IAM User Access Key Rotation

The challenge you are facing is common, but the bottleneck lies in how the AWS SDK credential provider chain handles caching.

By default, if an application initializes an AWS SDK client using static, long-term IAM credentials (from ~/.aws/credentials), the SDK caches those keys in memory indefinitely. Modifying the file on disk via a background process will not cause a running application to reload them; it will continue using the old, cached keys until the application restarts. The SDK only auto-refreshes credentials natively if they contain an expiration timestamp (like STS temporary tokens). Since you mentioned IAM Roles Anywhere (which is the official AWS best practice for external workloads) introduces too much operational overhead, the only bulletproof, AWS-supported pattern to refresh long-term IAM User keys without restarts is using the credential_process mechanism.

Here is how to implement it:

1. credential_process with Artificial Expiration

You can configure the ~/.aws/config file of your client applications to use an external script to fetch credentials from your central store (like AWS Secrets Manager or HashiCorp Vault):

[profile credential_rotation_profile]
credential_process = /opt/aws/bin/fetch_keys.sh

Your script (fetch_keys.sh) must return a specific JSON structure. Even though IAM User Access Keys do not natively expire, you must artificially inject an Expiration timestamp into the JSON payload:

{
  "Version": 1,
  "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",
  "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "Expiration": "2026-06-26T18:30:00Z"
}

Why this works: Modern AWS SDKs evaluate the Expiration field regardless of whether a SessionToken is present. Once the specified timestamp is reached, the SDK automatically re-executes your script on the next API call to fetch the updated keys entirely in the background—and it does so without requiring an application restart.

2. Backend Rotation Pattern (2-Key Rule)

An IAM User can hold a maximum of two active Access Keys. To prevent any downtime during rotation, your backend automation (e.g., an AWS Lambda function rotating the key in Secrets Manager) must follow this strict overlapping sequence:

1. Create Key B: Generate the new Access Key B in IAM. Both Key A and Key B are now concurrently active.
2. Update Central Store: Save Key B to AWS Secrets Manager.
3. Enforce a Grace Period: Wait longer than the TTL/Expiration window defined in your client applications (e.g., if your script sets a 1-hour expiration, wait 75 minutes). This ensures every distributed client instance has executed the script at least once and successfully transitioned to Key B.
4. Delete Key A: Safely deactivate and delete the old Key A from the IAM User. If you don't implement this overlapping grace period, clients that haven't refreshed their cache within the exact second of backend rotation will immediately fail.

Ultimately, if managing the 2-key lifecycle and secure secret retrieval on the client side becomes too complex, reconsidering IAM Roles Anywhere or switching to short-lived STS tokens assumed via a bastion IAM User are the only ways to eliminate long-term credential management entirely.

See also:

• https://repost.aws/selections/KPvXo5DmagRJ61BpYX9fiqhA/iam-access-key-rotation

• https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/centralize-iam-access-key-management-in-aws-organizations-by-using-terraform.html

• https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html