How do I troubleshoot Amazon EBS snapshot issues with Amazon Data Lifecycle Manager?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Verify that the lifecycle policy is active

If Amazon Data Lifecycle Manager doesn't create Amazon EBS snapshots, then complete the following steps to verify that the lifecycle policy is active:

1. Open the Amazon Elastic Compute Cloud (Amazon EC2) console.
2. In the navigation pane, choose Elastic Block Store, and then choose Lifecycle Manager.
3. Verify that the policy State is set to ENABLED. If the policy isn't set to ENABLED, choose Actions, and then choose Modify Snapshot Lifecycle Policy.
   Note: If the policy state is ERROR, then see the Resolve the lifecycle policy error state section.
4. Choose Enable policy, and then choose Update policy.

Note: After you create a lifecycle policy, it takes up to an hour to create snapshots. After at least an hour has passed, open the Amazon EC2 console. To verify that Amazon Data Lifecycle Manager created the snapshots, choose Elastic Block Store, and then choose Snapshots.

Verify AWS KMS key permissions for encrypted Amazon EBS snapshots

If the copied snapshot is encrypted, then you must have access to both the source and destination AWS Key Management Service (AWS KMS) key. The AWSDataLifecycleManagerDefaultRole must have permission to use the AWS KMS keys that encrypt volumes in both the source and destination Regions for cross-Region copy operations.

Resolve the lifecycle policy error state

Your lifecycle policy could be in the error state because of one or more of the following issues:

• There's a problem with your Amazon EBS resource tags.
• The Amazon Data Lifecycle Manager permissions are incorrect.
• The AWS Identity and Access Management (IAM) permissions are incorrect.

If you use a custom IAM role, then check whether the role has a trust relationship attached.

To get more information about what caused the error state, check Monitor Amazon Data Lifecycle Manager policies using Amazon EventBridge for DLM Policy State Change events.

Resolve a "Duplicate tag key" error on an Amazon EBS volume

If there are duplicate tags in your lifecycle policy, then you see an AWS CloudTrail event similar to the following example:

"errorCode": "Client.InvalidParameterValue",
"errorMessage": "Duplicate tag key 'Name' specified.",
"requestParameters": {
    "volumeId": "vol-############",
    "description": "Created for policy: policy-####schedule: First Schedule"
}

In the output example, the tag key Name is duplicated in the policy.

To resolve a "Duplicate tag key" error, complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Elastic Block Store, and then choose Lifecycle Manager.
3. Select your lifecycle policy.
4. Choose Actions, and then choose Modify Snapshot Lifecycle Policy.
5. In the Tag created EBS snapshots section, change the Key on the duplicated tag to a unique name.
6. Choose Update policy.

Resolve a "Tag is already defined in resource" error on an Amazon EBS volume

You can get an error when a tag that's defined in your lifecycle policy is already in use in a different lifecycle policy. You receive an error when both of the following conditions are true:

• The lifecycle policy is in the same account.
• The lifecycle policy targets the same Amazon EBS volume or Amazon EC2 instance.

When the tag is already defined, you see a CloudTrail event similar to the following example:

"errorCode": "Client.InvalidParameterCombination",
"errorMessage": "Tag (Name) is already defined in resource id vol-########.",
"requestParameters": {
    "CreateSnapshotsRequest": {
        "Description": "Created for policy: policy-#######schedule: Default Schedule",
        "InstanceSpecification": {
            "ExcludeBootVolume": false,
            "InstanceId": "i-#######"
        }
    }
}

An Amazon EBS volume can have more than one policy associated with it, but tags can't be duplicated across policies. For more information, see Considerations for snapshot lifecycle policies.

To resolve the error, complete the following steps:

1. View your lifecycle policies to identify the duplicated tag.
2. Create a new lifecycle policy with a different tag, or edit your current lifecycle policy to use a different tag.

Resolve a "Client.AuthFailure" error for encrypted snapshots

The "Client.AuthFailure" error occurs when the lifecycle policy or IAM user lacks permissions to access the AWS KMS key that encrypts the Amazon EBS volume or snapshot. The following example shows a "Client.AuthFailure" error caused by an inaccessible AWS KMS key:

"Client.AuthFailure",
"errorMessage": "The specified keyId arn:aws:kms:us-west-1:#############:key/4ad6a1d7-53ac-45a3-8f08-e6eccc948fdd is not accessible"

To set permissions for Amazon Data Lifecycle Manager, see AWS managed policies for Amazon Data Lifecycle Manager.

Check why a lifecycle policy creates Amazon EBS snapshots for certain volumes but not others

If a lifecycle policy creates Amazon EBS snapshots for certain volumes but not others, then the issue is at the volume level. To identify and resolve the issue, complete the following steps:

1. Compare the tags on a working volume against a non-working volume to identify differences. Check volume tags for inconsistencies. Amazon Data Lifecycle Manager targets Amazon EBS volumes based on exact tag matches. A trailing space, different casing such as Backup compared to backup, or a missing tag prevents Amazon Data Lifecycle Manager from targeting the volume.
2. Verify the volume state. Amazon Data Lifecycle Manager can snapshot only Amazon EBS volumes in the available or in-use state. If a volume is in the creating, deleting, or error state, then Amazon Data Lifecycle Manager skips the volume without notification.
3. Verify AWS KMS key permissions for encrypted volumes. If you encrypt certain Amazon EBS volumes with a customer managed AWS KMS key, then you must grant the AWSDataLifecycleManagerDefaultRole, kms:CreateGrant, kms:GenerateDataKey, and kms:Decrypt permissions on that specific key. Amazon EBS volumes that use the default AWS managed key aren't affected by this issue. For more information, see Control access to Amazon Data Lifecycle Manager using IAM.
4. Check for recently tagged volumes. Amazon Data Lifecycle Manager evaluates destination Amazon EBS volumes at run time, not continuously. If you tagged volumes after the last policy run, then Amazon Data Lifecycle Manager picks them up on the next scheduled run.
5. Check the concurrent Amazon EBS snapshot quotas. There's a quota on concurrent pending snapshots per volume. If you tagged Amazon EBS volumes after the last policy run, then Amazon Data Lifecycle Manager picks them up on the next scheduled run.

Check why snapshots aren't automatically deleted

If Amazon Data Lifecycle Manager creates Amazon EBS snapshots but doesn't delete them automatically, then complete the following steps to identify and resolve the issue:

1. Verify retention settings. Check whether your policy uses count-based or age-based retention. For count-based retention, such as a retain count of 7, Amazon Data Lifecycle Manager deletes snapshots only after the count exceeds the configured number. For age-based retention, such as a retain period of 7 days, Amazon Data Lifecycle Manager deletes snapshots older than the configured period on the next policy run. For more information, see Considerations for snapshot lifecycle policies.

   To verify the retention configuration for your Amazon EBS snapshot policy, run the following get-lifecycle-policy AWS CLI command:

   aws dlm get-lifecycle-policy \
       --policy-id POLICY_ID \
       --query "Policy.PolicyDetails.Schedules[*].RetainRule"

   Note: Replace POLICY_ID with your lifecycle policy ID.

2. Check for modified snapshot tags. Amazon Data Lifecycle Manager uses the aws:dlm:lifecycle-policy-id tag to track its managed snapshots. If you remove or modify the tag, then Amazon Data Lifecycle Manager no longer recognizes those Amazon EBS snapshots as managed and won't delete them.

3. Check for snapshot locks. If you lock snapshots, then Amazon Data Lifecycle Manager can't delete them regardless of the retention policy. Check whether your snapshots have a lock applied before you investigate other causes.

Resolve encrypted snapshots deleted despite a configured retention period for cross-Region copies

When Amazon Data Lifecycle Manager copies an encrypted snapshot across Regions, it requires ongoing AWS KMS permissions. These permissions must exist in both the source and destination Regions to manage the snapshot lifecycle. If the AWS KMS key policies are too restrictive, then Amazon Data Lifecycle Manager can perform the initial copy but can't maintain the snapshot afterward. Amazon Data Lifecycle Manager then treats the snapshot as unmanageable and deletes it.

To resolve this issue, verify that the lifecycle policy's IAM role has the required AWS KMS permissions in both the source and destination Regions. Complete the following steps:

1. Open the AWS KMS console.
2. In the navigation pane, choose Customer managed keys.
3. Select your AWS KMS key.
4. On the Key policy tab, choose Edit.
5. Verify that the key policy lists the lifecycle policy's IAM role, such as AWSDataLifecycleManagerDefaultRole, as a key user with the following permissions: kms:CreateGrant
   kms:Decrypt
   kms:DescribeKey
   kms:Encrypt
   kms:GenerateDataKey
   kms:ReEncrypt*
6. If the role isn't listed, then add it as a key user and choose Save changes.
7. Repeat these steps for the AWS KMS key in the destination Region.

To set permissions for IAM users to use Amazon Data Lifecycle Manager, see Control access to Amazon Data Lifecycle Manager using IAM.

Related information

Troubleshoot Amazon Data Lifecycle Manager issues