By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Backup with Amazon Elastic File System: backup consistency solutions

0

The Amazon EFS documentation states:

Backup consistency

Amazon EFS is designed to be highly available. You can access and modify your EFS file systems while your backup is occurring in AWS Backup. However, inconsistencies, such as duplicated, skewed, or excluded data, can occur if you make modifications to your file system while the backup is occurring. These modifications include write, rename, move, or delete operations. To ensure consistent backups, we recommend that you pause applications or processes that are modifying the file system for the duration of the backup process. Or, schedule your backups to occur during periods when the file system is not being modified.

https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html#how-backup-works

I'm specifically interested in the comment: However, inconsistencies, such as duplicated, skewed, or excluded data, can occur if you make modifications to your file system while the backup is occurring..

We have operations running 24/7 on these file systems. It is not feasible to block access.

My question is how does the AWS Backup design handles these potential inconsistencies?

  • Are these inconsistencies resolved in a subsequent backup or are they retained in an inconsistent state forever?
  • Are there specific activities that can be triggered on the filesystem that would cause the files to be backed up safely, such as touch?
  • Do you recommend instead that the backup is fully restored and then affected files are identified and resolved? I'd like to avoid over-engineering a solution
Topics
Storage
Tags
Amazon Elastic File SystemAWS Backup
Language
English
Stephen
asked 3 months ago157 views
1 Answer
1

Hello Stephen,

You're right, pausing applications for backups isn't ideal with 24/7 operations. Here's how AWS Backup with EFS handles potential inconsistencies:

AWS Backup for Amazon EFS may encounter inconsistencies if the file system is modified during a backup. These inconsistencies (like duplicated or missing data) are specific to that snapshot and won't be automatically resolved.

To Solve this:

  • Schedule backups during low-activity periods if possible.
  • Regularly restore and validate backups to ensure data integrity.
  • Implement application-level strategies to minimize modifications during backups.

Go through This Link: https://aws.amazon.com/getting-started/hands-on/amazon-efs-backup-and-restore-using-aws-backup/

profile picture
EXPERT
NARRAVULA MUNISAI TEJA
answered 3 months ago
  • Stephen
    3 months ago

    Thanks, I am aware of these but I'd like a slightly more nuanced answer if possible.

    Let me be clear. The 24/7 activities I am referring to are long-running processes, often spanning multiple days, of batch processing by analysis engines. These are equally weighted read-write operations. We don't want to include substantial dead periods or miss regular backups.

    You mention that the inconsistencies are "specific to that snapshot". Does that mean that the next snapshot will have a correct backup of the affected changes, assuming no further modifications, or will the data still be in the failed state if restoring from future backups?

    Note that "regularly restoring and validating backups" only demonstrates the issue, it doesn't actually resolve anything. I'm looking for solutions that resolve it after the issue has occured.

    The other options would basically entail duplicating and verifying substantial volumes of data. We can do this, but I'm wondering if there's a lighter touch approach.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content