Troubleshooting and preventing undeletable recovery points in AWS Backup Compliance mode vaults

Introduction

A financial services company recently contacted AWS Enterprise Support with an urgent concern. Their cloud infrastructure team had configured AWS Backup Vault Lock in Compliance mode on several production vaults to meet regulatory data retention requirements.

Several months later, the team realized that they configured their vaults without a MaxRetentionDays value. Certain recovery points had retention periods set to Forever, and the grace time period expired. AWS Backup didn't automatically delete these recovery points, and the team couldn't delete them, even as the AWS account root user.

This article walks through the troubleshooting approach that you can use, the decision framework, and the preventive measures that you can implement.

Key concepts: Grace time and maximum retention

There are two critical Compliance mode vault lock parameters at the center of the issue, Grace time and the maximum Retention period.

Grace time (ChangeableForDays) is a configurable cooling-off period with a 72-hour minimum. During the grace time, you can update or remove the vault lock. After the grace time expires, the lock becomes permanent. For more information, see AWS Backup Vault Lock.
Important: If you incorrectly configured your vault lock, then engage with AWS Support during the grace time. You can correct your configuration only during this period.

Maximum retention period (MaxRetentionDays) is an optional vault lock parameter that sets the maximum number of days that AWS Backup Vault Lock retains recovery points. If you don't set a MaxRetentionDays value, then the following conditions are true:

• If a recovery point's retention period is set to Forever at backup creation, then AWS Backup retains it indefinitely.

• No one can delete your recovery points.

The MaxRetentionDays parameter is separate from the recovery point retention that you configure in the lifecycle rules of your AWS Backup plan. It doesn't retroactively change the retention of recovery points.

Step 1: Check the vault lock state

Run the following describe-backup-vault AWS Command Line Interface (AWS CLI) command:

aws backup describe-backup-vault \
     --backup-vault-name example-vault-name

Note: Replace example-vault-name with the name of your vault.

Use the following table to interpret the fields in the output:

Field	Value	Meaning
Locked	true / false	This field shows whether vault lock is active.
LockDate	Date in the future	During this time period, the grace time is active, and you can update or remove the lock.
LockDate	Date in the past	During this time period, the grace time is expired, and the lock is permanent.
MaxRetentionDays	Numeric value	This field shows the maximum number of days that AWS Backup enforces retention on new jobs.

If LockDate is present, then the vault is in Compliance mode. If Locked is true but LockDate is absent, then the vault is in Governance mode and you can move it with AWS Identity and Access Management (IAM) permissions. The following scenarios apply only to Compliance mode vaults. For more information, see Review a backup vault for its AWS Backup Vault Lock configuration.

In the following example output, the vault is in Compliance mode:

{
     "BackupVaultName": "production-compliance-vault",
     "NumberOfRecoveryPoints": 47,
     "Locked": true,
     "MinRetentionDays": 7,
     "MaxRetentionDays": 365,
     "LockDate": "2024-01-18T08:30:00.000000+00:00"
 }

Step 2: Evaluate your options

Scenario A: The grace time is still active and the LockDate is in the future

This is the best-case scenario. You can still update or remove the vault lock.

To remove the vault lock, run the following delete-backup-vault-lock-configuration AWS CLI command:

aws backup delete-backup-vault-lock-configuration \
     --backup-vault-name example-vault-name

Note: Replace example-vault-name with the name of your vault.

To modify the vault lock, run the following put-backup-vault-lock-configuration AWS CLI command:

aws backup put-backup-vault-lock-configuration \
     --backup-vault-name example-vault-name \
     --changeable-for-days 3 \
     --min-retention-days 7 \
     --max-retention-days 365

Note: Replace example-vault-name with the name of your vault.

Scenario B: The grace period is expired and MaxRetentionDays is finite

In this example scenario, the vault lock is permanent, but each recovery point automatically expires. AWS Backup deletes each recovery point at the end of its retention period.

Use the AWS Management Console or AWS CLI to note your recovery point expiration dates. Or, redirect new AWS Backup jobs to a different vault. For instructions, see Step 3: Redirect your AWS Backup plans for immediate remediation.

To use the AWS CLI to check when recovery points expire, run the following list-recovery-points-by-backup-vault command:

aws backup list-recovery-points-by-backup-vault \
     --backup-vault-name example-vault-name \
     --query 'RecoveryPoints[].{ResourceArn:ResourceArn,CreationDate:CreationDate,CompletionDate:CompletionDate,Lifecycle:Lifecycle}'

Note: Replace example-vault-name with the name of your vault.

Then, wait for recovery points to reach their retention expiration dates.

If you create recovery points before your vault locks and configure their total retention periods to Forever, then the following conditions are true about your recovery points:

• They have an empty (null) Lifecycle, so no DeleteAfterDays is set and the recovery point has no expiration date.

Scenario C: The Grace period is expired and there are no MaxRetentionDays, or retention is set to Forever

This is the most constrained scenario. The Support team, the AWS Backup team, and the root user can't delete recovery points with retention set to Forever from a Compliance mode vault after the grace time period expires.

You have the following options:

• Stop creating new recovery points in your vault.

• Monitor the ongoing storage costs for your locked recovery points.

• As a last resort, close your AWS account. After the 90-day suspension period, AWS deletes the vault contents for deleted accounts.
  Important: Close your account as a last resort only. Before you do, migrate all your workloads and resources to a new account.

Step 3: Redirect your AWS Backup plans for immediate remediation

Important: It's a best practice to redirect your AWS Backup plans so that you don't create new recovery points in the problematic vault.

To redirect your AWS Backup plans, complete the following steps:

1. To create a new AWS Backup vault with an appropriate vault lock configuration, including MaxRetentionDays, run the following AWS CLI commands.

   First, run the create-backup-vault command to create the vault:

   aws backup create-backup-vault \
        --backup-vault-name new-compliant-vault \
        --encryption-key-arn arn:aws:kms:us-east-1:111122223333:key/your-kms-key-id
   

   Note: Replace us-east-1 with your AWS Region, 111222333 with your AWS account ID, and your-kms-key-id with the ID of your AWS Key Management Service Customer Managed Key.

   Then, run the put-backup-vault-lock-configuration command to apply the vault lock with MaxRetentionDays:

   aws backup put-backup-vault-lock-configuration \
        --backup-vault-name new-compliant-vault \
        --changeable-for-days 3 \
        --min-retention-days 7 \
        --max-retention-days 365
   

2. Update your AWS Backup plans to target the new vault.

3. To identify other vaults with the same misconfiguration, run the following script to check all vaults in every AWS Region where you activated AWS Backup:

   #!/bin/bash
    echo "Region | Vault Name | Locked | LockDate | MaxRetentionDays"
    echo "------ | ---------- | ------ | -------- | ----------------"
    
   for region in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
        for vault in $(aws backup list-backup-vaults --region "$region" --query 'BackupVaultList[].BackupVaultName' --output text 2>/dev/null); do
            result=$(aws backup describe-backup-vault --backup-vault-name "$vault" --region "$region" \
                --query '{Locked:Locked, LockDate:LockDate, MaxRetentionDays:MaxRetentionDays}' \
                --output json 2>/dev/null)
            
           locked=$(echo "$result" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('Locked','N/A'))" 2>/dev/null)
            lockdate=$(echo "$result" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('LockDate','Not set'))" 2>/dev/null)
            maxret=$(echo "$result" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('MaxRetentionDays','NOT SET'))" 2>/dev/null)
            
           if [ "$locked" = "True" ] && [ "$maxret" = "None" ]; then
                echo " $region | $vault | $locked | $lockdate | NOT SET (RISK)"
            else
                echo "   $region | $vault | $locked | $lockdate | $maxret"
            fi
        done
    done
   

If a vault shows Locked: True with MaxRetentionDays: NOT SET, then set the maximum retention value.

Step 4: Enforce retention periods with SCPs

To implement an SCP that denies vault lock creation without a MaxRetentionDays value, run the following script:

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "DenyVaultLockWithoutMaxRetention",
             "Effect": "Deny",
             "Action": [
"backup:PutBackupVaultLockConfiguration",
"backup:CreateLogicallyAirGappedBackupVault"
],
             "Resource": "*",
             "Condition": {
                 "NumericGreaterThan": {
                     "backup:MaxRetentionDays": "1825"
                 }
             }
         },
         {
             "Sid": "DenyVaultLockWithNullMaxRetention",
             "Effect": "Deny",
             "Action": [
"backup:PutBackupVaultLockConfiguration",
"backup:CreateLogicallyAirGappedBackupVault"
],
             "Resource": "*",
             "Condition": {
                 "Null": {
                     "backup:MaxRetentionDays": "true"
                 }
             }
         }
     ]
 }

After you run the preceding script, you can't create a vault lock without MaxRetentionDays. You also can't create a vault lock with a MaxRetentionDays value that's greater than 1,825 days, or 5 years. In the preceding script, the MaxRetentionDays value is set to 1,825 days. Adjust the MaxRetentionDays value to match your requirements.

Note: SCPs apply only to AWS Organizations member accounts, not the management account. For more information, see How to enforce Minimum and Maximum Retention Periods for AWS Backup Vault Lock using SCPs.

Conclusion

The grace time period that you set is your safety net. After it expires, your vault lock configuration is permanent.

To use Compliance mode for vault locks and avoid permanently undeletable recovery points, take the following actions:

1. Understand vault lock states.

2. Audit your existing vaults.

3. Use SCPs to implement organizational guardrails.

If you're unsure about your current configurations, use the preceding method to audit your vaults. For more information, contact Support.

Related information

Security in AWS Backup

About the author

Ankush Singh is a Cloud Support Engineer at AWS who specializes in helping customers with their storage and AWS Backup solutions in the cloud. He has over 12 years of experience in IT and is passionate about the benefits of generative AI technologies to solve complex business challenges. Outside of work, Ankush enjoys playing cricket, video games, cooking, and exploring countries on vacation with his family.