Why did my Secrets Manager Lambda rotation fail?

Short description

In December 2024, Secrets Manager changed how it validates entities that call the PutSecretValue API action as part of the rotation process. This change affects both cross-account and assumed role rotation.

The function must pass a rotation token so that Secrets Manager can validate the entity's identity. Lambda rotation functions can then call PutSecretValue.

If you didn't updated your rotation function code before December 2024, then it's a best practice to update it. Check AWS CloudTrail for a RotationFailed event with the message, "Pending secret version VERSION_ID for Secret SECRET_ARN wasn't created by Lambda LAMBDA_ARN. Remove the AWSPENDING staging label and restart rotation". If you find the RotationFailed event, then update your Lambda function to use the RotationToken parameter.

Note: If you don't use cross-account rotation or an assumed role, then no changes are required.

Resolution

Download the Lambda function code

Complete the following steps:

1. Open the Lambda console.
2. In the navigation pane, choose Functions.
3. For Function name, select your Lambda function that rotates the secret.
4. For Download, choose one of the following options:
   Function code .zip
   AWS SAM file
   Both
5. Choose OK to save the function on your local machine.

Edit the Lambda function code

Open your code editor, and edit your Lambda function code. The following examples are code edits that you might need to make.

Edit lambda_handler code

Include the rotation_token parameter in the create_secret step for cross-account rotation:

def lambda_handler(event, context):
    """Secrets Manager Rotation Template

    This is a template for creating an AWS Secrets Manager rotation lambda

    Args:
        event (dict): Lambda dictionary of event parameters. These keys must include the following:
            - SecretId: The secret ARN or identifier
            - ClientRequestToken: The ClientRequestToken of the secret version
            - Step: The rotation step (one of createSecret, setSecret, testSecret, or finishSecret)
            - RotationToken: the rotation token to put as parameter for PutSecretValue call

        context (LambdaContext): The Lambda runtime information

    Raises:
        ResourceNotFoundException: If the secret with the specified arn and stage does not exist

        ValueError: If the secret is not properly configured for rotation

        KeyError: If the event parameters do not contain the expected keys

    """
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']
    # Add the rotation token
    rotation_token = event['RotationToken']

    # Setup the client
    service_client = boto3.client('secretsmanager', endpoint_url=os.environ['SECRETS_MANAGER_ENDPOINT'])

    # Make sure the version is staged correctly
    metadata = service_client.describe_secret(SecretId=arn)
    if not metadata['RotationEnabled']:
        logger.error("Secret %s is not enabled for rotation" % arn)
        raise ValueError("Secret %s is not enabled for rotation" % arn)
    versions = metadata['VersionIdsToStages']
    if token not in versions:
        logger.error("Secret version %s has no stage for rotation of secret %s." % (token, arn))
        raise ValueError("Secret version %s has no stage for rotation of secret %s." % (token, arn))
    if "AWSCURRENT" in versions[token]:
        logger.info("Secret version %s already set as AWSCURRENT for secret %s." % (token, arn))
        return
    elif "AWSPENDING" not in versions[token]:
        logger.error("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn))
        raise ValueError("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn))
    # Use rotation_token
    if step == "createSecret":
        create_secret(service_client, arn, token, rotation_token)

    elif step == "setSecret":
        set_secret(service_client, arn, token)
    
    elif step == "testSecret":
        test_secret(service_client, arn, token)
        
    elif step == "finishSecret":
        finish_secret(service_client, arn, token)
        
    else:
        raise ValueError("Invalid step parameter")

Edit create_secret code

Update the create_secret function to accept and use the rotation_token parameter:

# Add rotation_token to the function
def create_secret(service_client, arn, token, rotation_token):
"""Create the secret

This method first checks for the existence of a secret for the passed in token. If one does not exist, it will generate a
new secret and put it with the passed in token.

Args:
service_client (client): The secrets manager service client

arn (string): The secret ARN or other identifier

token (string): The ClientRequestToken associated with the secret version

rotation_token (string): the rotation token to put as parameter for PutSecretValue call

Raises:
ResourceNotFoundException: If the secret with the specified arn and stage does not exist

"""
# Make sure the current secret exists
service_client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")

# Now try to get the secret version, if that fails, put a new secret
try:
service_client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")
logger.info("createSecret: Successfully retrieved secret for %s." % arn)
except service_client.exceptions.ResourceNotFoundException:
# Get exclude characters from environment variable
exclude_characters = os.environ['EXCLUDE_CHARACTERS'] if 'EXCLUDE_CHARACTERS' in os.environ else '/@"\'\\'
# Generate a random password
passwd = service_client.get_random_password(ExcludeCharacters=exclude_characters)

# Put the secret, using rotation_token
service_client.put_secret_value(SecretId=arn, ClientRequestToken=token, SecretString=passwd['RandomPassword'], VersionStages=['AWSPENDING'], RotationToken=rotation_token)
logger.info("createSecret: Successfully put secret for ARN %s and version %s." % (arn, token))

Upload the updated Lambda function code

After you update your Lambda function code, upload the code to rotate your secret.

Related information

Rotation by Lambda function

My Lambda rotation function called a second function to rotate a Secrets Manager secret but it failed. Why did the second Lambda function fail?