Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Python script works to copy most recent snapshots, but then also tries to copy (2) more snapshots which we do not want it to do.

0

Hi re:Post!
This question is related to (2) previous posts, "Permission needed for Role to Use KMS key as part of a Lambda function" and "Python code deleted wrong snapshots and tried to copy wrong snapshots".

The python script no longer errors when the Lambda Role's Policy accesses and the customer made KMS key, "rds-cross-region-replication". Awesome! And thank you Riku!

The python script successfully deletes the latest (2) snapshots in Oregon, "us-west-2" manually generated snapshots via their respective database names.

And then the python script successfully copies the most recent (2) snapshots from Oregon, "us-west-2", system generated snapshots.

However, the python script then tries to copy (2) more snapshots ie the 2nd most recent (2) snapshots from Oregon, "us-west-2", system generated snapshots, which we do not want it to do. We want it to only copy the most recent and not any others.
I'm trying to figure out why it is doing this.

To summarize, below is the correct copy behavior.

The python script successfully copies the (2) most recent snapshots found in "us-west-2" system generated snapshots based on the "db_names" variable in script. "db_names" lists (2) databases.

  1. Copying snapshot "rds:db-drtest20250410-2025-08-22-05-10" to "copy-rds-db-drtest20250410-2025-08-22" Successfully initiated copy of rds:db-drtest20250410-2025-08-22-05-10 to copy-rds-db-drtest20250410-2025-08-22

  2. Copying snapshot rds:db-drtest20240314-2025-08-22-05-05 to copy-rds-db-drtest20240314-2025-08-22 Successfully initiated copy of rds:db-drtest20240314-2025-08-22-05-05 to copy-rds-db-drtest20240314-2025-08-22

Then it tries to do the same for snapshots from August 21st at "us-west-2" system generated and also rename them the same as 1) and 2) above, so it errors because of same name of the (2) successful and correct snapshot copies. We don't want the script to try to also copy the 2nd most recent copies, ie the August 21st ones. Just the most recent and stop.

I'm just not figuring out why, it's late on Friday ;) ....

Below is the runtime success and error stack.

At at the bottom is the python script.

Thank you for your time and help!

Best Regards,

Donald

Full Runtime output:

Function Logs:
START RequestId: f88cf2a7-4156-4c92-9c31-98b4ec6ccb8f Version: $LATEST
Deleting snapshot: copy-rds-db-drtest20250410-2025-08-20
Successfully deleted snapshot: copy-rds-db-drtest20250410-2025-08-20
Deleting snapshot: copy-rds-db-drtest20240314-2025-08-20
Successfully deleted snapshot: copy-rds-db-drtest20240314-2025-08-20

Copying snapshot rds:db-drtest20250410-2025-08-22-05-10 to copy-rds-db-drtest20250410-2025-08-22
Successfully initiated copy of rds:db-drtest20250410-2025-08-22-05-10 to copy-rds-db-drtest20250410-2025-08-22

Copying snapshot rds:db-drtest20250410-2025-08-21-05-10 to copy-rds-db-drtest20250410-2025-08-22
Error copying snapshot rds:db-drtest20250410-2025-08-21-05-10: An error occurred (DBSnapshotAlreadyExists) when calling the CopyDBSnapshot operation: Cannot copy the snapshot because a snapshot with the identifier arn:aws:rds:us-west-2:910286192445:snapshot:copy-rds-db-drtest20250410-2025-08-22 already exists.

Copying snapshot rds:db-drtest20240314-2025-08-22-05-05 to copy-rds-db-drtest20240314-2025-08-22
Successfully initiated copy of rds:db-drtest20240314-2025-08-22-05-05 to copy-rds-db-drtest20240314-2025-08-22

Copying snapshot rds:db-drtest20240314-2025-08-21-05-05 to copy-rds-db-drtest20240314-2025-08-22
Error copying snapshot rds:db-drtest20240314-2025-08-21-05-05: An error occurred (DBSnapshotAlreadyExists) when calling the CopyDBSnapshot operation: Cannot copy the snapshot because a snapshot with the identifier arn:aws:rds:us-west-2:910286192445:snapshot:copy-rds-db-drtest20240314-2025-08-22 already exists.

END RequestId: f88cf2a7-4156-4c92-9c31-98b4ec6ccb8f
REPORT RequestId: f88cf2a7-4156-4c92-9c31-98b4ec6ccb8f  Duration: 4683.09 ms    Billed Duration: 4684 ms    Memory Size: 128 MB Max Memory Used: 89 MB  Init Duration: 328.07 ms

Python script for "Lambda-Function-RDS-Snapshot-Management":

import boto3
import os
from datetime import datetime

# Define regions
# SOURCE_REGION = "us-west-1"  # N. California
SOURCE_REGION = "us-west-2"  # Oregon
DEST_REGION = "us-west-2"    # Oregon
NUM_SNAPSHOTS = 2            # Number of snapshots to process

def lambda_handler(event, context):
# Create RDS clients for both regions
    source_rds = boto3.client('rds', region_name=SOURCE_REGION)
    dest_rds = boto3.client('rds', region_name=DEST_REGION)

# Step 1: Delete oldest manual snapshots in destination region
    delete_oldest_snapshots(dest_rds, NUM_SNAPSHOTS)

# Step 2: Copy latest system snapshots from source to destination
    copy_latest_snapshots(source_rds, dest_rds, NUM_SNAPSHOTS)

    return {
        'statusCode': 200,
        'body': f'Attempted to process {NUM_SNAPSHOTS} snapshots'
    }

def delete_oldest_snapshots(dest_rds, count):

    db_names = [
        "db-drtest20250410", "db-drtest20240314"
    ]

# Get all manual snapshots in destination region
    for db_name in db_names:
        response = dest_rds.describe_db_snapshots(
            DBInstanceIdentifier=db_name,
            SnapshotType='manual'
        )

    # Sort snapshots by creation time (oldest first)
        snapshots = sorted(response['DBSnapshots'], key=lambda s: s['SnapshotCreateTime'])

    # Delete the oldest 'count' snapshots
        for i, snapshot in enumerate(snapshots):
            if i >= count:
                break

            snapshot_id = snapshot['DBSnapshotIdentifier']
            print(f"Deleting snapshot: {snapshot_id}")

            try:
                dest_rds.delete_db_snapshot(DBSnapshotIdentifier=snapshot_id)
                print(f"Successfully deleted snapshot: {snapshot_id}")
            except Exception as e:
                print(f"Error deleting snapshot {snapshot_id}: {str(e)}")

def copy_latest_snapshots(source_rds, dest_rds, count):
    # Define the database names based on your example
    #  "amgen", "alpine"
    db_names = [
        "db-drtest20250410", "db-drtest20240314"
    ]

    for db_name in db_names:
    # Get all automated snapshots in source region
        response = source_rds.describe_db_snapshots(
            DBInstanceIdentifier=db_name,
            SnapshotType='automated'
        )

        # Sort snapshots by creation time (newest first)
        snapshots = sorted(response['DBSnapshots'], 
            key=lambda s: s['SnapshotCreateTime'], 
            reverse=True
        )

        # Get today's date for naming
        today = datetime.now().strftime("%Y-%m-%d")

        # Copy the newest 'count' snapshots
        for i, snapshot in enumerate(snapshots[:count]):
            if i >= count:
                break

            source_snapshot_id = snapshot['DBSnapshotIdentifier']

            # Create target snapshot name
            target_snapshot_id = f"copy-rds-{db_name}-{today}"

            print(f"Copying snapshot {source_snapshot_id} to {target_snapshot_id}")

            try:
            # Create ARN for source snapshot
                source_arn = snapshot['DBSnapshotArn']

            # Copy the snapshot
                dest_rds.copy_db_snapshot(
                    SourceDBSnapshotIdentifier=source_arn,
                    TargetDBSnapshotIdentifier=target_snapshot_id,
                    KmsKeyId="arn:aws:kms:us-west-2:910286192445:key/mrk-6dae29119b094afaa3b9ed67c781ab3c",
                    SourceRegion=SOURCE_REGION
                )
                print(f"Successfully initiated copy of {source_snapshot_id} to {target_snapshot_id}")
            except Exception as e:
                print(f"Error copying snapshot {source_snapshot_id}: {str(e)}")

end of post...

Topics
ServerlessCompute
Tags
AWS Lambda
Language
English

asked a year ago143 views

2 Answers
1
Accepted Answer

Hello.

The following if statement retrieves the number of snapshots specified by "count" from the "snapshots" variable.
Your code currently has "NUM_SNAPSHOTS" set to 2, which means it will copy two of the most recent snapshots.

        # Copy the newest 'count' snapshots
        for i, snapshot in enumerate(snapshots[:count]):
            if i >= count:
                break

If you only want to copy the most recent one, there is no need for "for i, snapshot in enumerate(snapshots[:count]):".
As stated in the AI ​​automated response from re:Post Agent, you can copy the first latest snapshot by getting "snapshots[0]".
So, the "copy_latest_snapshots" function would look like this:

def copy_latest_snapshots(source_rds, dest_rds):
    # Define the database names based on your example
    #  "amgen", "alpine"
    db_names = [
        "db-drtest20250410", "db-drtest20240314"
    ]

    for db_name in db_names:
    # Get all automated snapshots in source region
        response = source_rds.describe_db_snapshots(
            DBInstanceIdentifier=db_name,
            SnapshotType='automated'
        )

        # Sort snapshots by creation time (newest first)
        snapshots = sorted(response['DBSnapshots'], 
            key=lambda s: s['SnapshotCreateTime'], 
            reverse=True
        )

        # Get today's date for naming
        today = datetime.now().strftime("%Y-%m-%d")

        # Copy the newest 'count' snapshots
        if snapshots:
            snapshot = snapshots[0]

            source_snapshot_id = snapshot['DBSnapshotIdentifier']

            # Create target snapshot name
            target_snapshot_id = f"copy-rds-{db_name}-{today}"

            print(f"Copying snapshot {source_snapshot_id} to {target_snapshot_id}")

            try:
            # Create ARN for source snapshot
                source_arn = snapshot['DBSnapshotArn']

            # Copy the snapshot
                dest_rds.copy_db_snapshot(
                    SourceDBSnapshotIdentifier=source_arn,
                    TargetDBSnapshotIdentifier=target_snapshot_id,
                    KmsKeyId="arn:aws:kms:us-west-2:910286192445:key/mrk-6dae29119b094afaa3b9ed67c781ab3c",
                    SourceRegion=SOURCE_REGION
                )
                print(f"Successfully initiated copy of {source_snapshot_id} to {target_snapshot_id}")
            except Exception as e:
                print(f"Error copying snapshot {source_snapshot_id}: {str(e)}")

By the way, if you want to create a specified number of copies from the latest snapshot, you will need to modify "target_snapshot_id = f"copy-rds-{db_name}-{today}"" so that it becomes a unique value.
For example, if you include the date and time in the snapshot name, there will be no duplication unless they are executed at the same time.

today = datetime.now().strftime("%Y-%m-%d-%H%M%S")

You could also add a process to check for duplicate names.
You can also add a number to the end of the snapshot name if there are duplicates by adding a process to check for duplicates as shown below.

def copy_latest_snapshots(source_rds, dest_rds, count):
    # Define the database names based on your example
    #  "amgen", "alpine"
    db_names = [
        "db-drtest20250410", "db-drtest20240314"
    ]

    for db_name in db_names:
    # Get all automated snapshots in source region
        response = source_rds.describe_db_snapshots(
            DBInstanceIdentifier=db_name,
            SnapshotType='automated'
        )

        # Sort snapshots by creation time (newest first)
        snapshots = sorted(response['DBSnapshots'], 
            key=lambda s: s['SnapshotCreateTime'], 
            reverse=True
        )

        # Get today's date for naming
        today = datetime.now().strftime("%Y-%m-%d")

        # Copy the newest 'count' snapshots
        for i, snapshot in enumerate(snapshots[:count]):
            if i >= count:
                break

            source_snapshot_id = snapshot['DBSnapshotIdentifier']

            # Create target snapshot name
            base_snapshot_id = f"copy-rds-{db_name}-{today}"
            target_snapshot_id = f"copy-rds-{db_name}-{today}"

            # Name Duplicate check
            index = 1
            while True:
                existing_snapshots = rds.describe_db_snapshots(
                    DBSnapshotIdentifier=target_snapshot_id,
                    SnapshotType='manual'
                )
                if not existing_snapshots["DBSnapshots"]:
                    break
                target_snapshot_id = f"{base_snapshot_id}-{index}"
                index += 1

            print(f"Copying snapshot {source_snapshot_id} to {target_snapshot_id}")

            try:
            # Create ARN for source snapshot
                source_arn = snapshot['DBSnapshotArn']

            # Copy the snapshot
                dest_rds.copy_db_snapshot(
                    SourceDBSnapshotIdentifier=source_arn,
                    TargetDBSnapshotIdentifier=target_snapshot_id,
                    KmsKeyId="arn:aws:kms:us-west-2:910286192445:key/mrk-6dae29119b094afaa3b9ed67c781ab3c",
                    SourceRegion=SOURCE_REGION
                )
                print(f"Successfully initiated copy of {source_snapshot_id} to {target_snapshot_id}")
            except Exception as e:
                print(f"Error copying snapshot {source_snapshot_id}: {str(e)}")
EXPERT

answered a year ago

  • DC
    a year ago

    Thank you again Riku for all your help! My test lambda function/python script works like a charm. It deletes the oldest snapshot for each database in us-west-2 (manual) and then copies the most recent snapshot in us-west-2 (system) over to us-west-2 (manual) with the correct renaming format! Next, I will move on from the testing stage to trying this out in production after changing the source from us-west-2 (system generated, Oregon) to us-west-1 (system generated, N. California) and using other databases. Best Regards, Donald

0

The issue in your Lambda function is related to how you're processing snapshots in the copy_latest_snapshots function. The problem occurs because you're retrieving multiple snapshots for each database and then trying to copy all of them up to the count limit, but you're using the same target snapshot name for all copies from the same database.

Here's what's happening:

  1. For each database in db_names, you retrieve all automated snapshots
  2. You sort them by creation time (newest first)
  3. You then try to copy the newest count (2) snapshots for each database
  4. However, you're using the same target name (copy-rds-{db_name}-{today}) for all snapshots from the same database

Since you only want to copy the single most recent snapshot for each database, you need to modify your code to only process the first snapshot in the sorted list.

Change this part of your code:

# Copy the newest 'count' snapshots
for i, snapshot in enumerate(snapshots[:count]):
if i >= count:
break

To this:

# Copy only the newest snapshot
if snapshots:
snapshot = snapshots[0]  # Get only the most recent snapshot

This way, for each database, you'll only copy the single most recent snapshot instead of trying to copy multiple snapshots with the same target name. This will prevent the "DBSnapshotAlreadyExists" error you're seeing.

Additionally, make sure your NUM_SNAPSHOTS variable is set correctly. Since you're only copying one snapshot per database and you have two databases, you might want to set it to 1 to be clearer about your intent.
Sources
Copied snapshot is being deleted after creation by DLM | AWS re:Post
How to automate daily delete snapshots and automate daily copy snapshots between regions | AWS re:Post
How to automate coping snapshots from N.California to Oregon? | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content