DataSync symlink target file from NFS server to EFS

0

I am trying to use DataSync to copy my file from on-premise(NFS 3) to EFS. It says symbolic links are preserved when copying from NFS file server and EFS,(https://docs.aws.amazon.com/datasync/latest/userguide/special-files-copied.html#special-files-copied-symbolic-links). But I want the DataSync to copy the target file of the symlink to the EFS. I've looked into options or datasync cli, but couldn't find a solution.

Any options to suggest?(I have to keep the symlink, but target file destinations are spread out, which makes it hard to get a pattern of them)

asked 3 months ago184 views
3 Answers
3
Accepted Answer

If you need to actually follow the symbolic link and copy the data, you can always use rsync with specific options to handle symlinks effectively if AWS DataSync does not meet your requirements.

The -L and -K options in rsync control how symlinks are handled during the synchronization process:

  • -L or --copy-links: This option tells rsync to transform the symlinks into referent files/dirs. In other words, if you have a symlink that points to a file or directory, it will copy the actual file or directory that the symlink points to insteaf of copying the symlink itself.

  • -K or --keep-dirlinks: This option is used with -L to avoid duplicating the referent of a symlinked directory inside a copied tree again at the location of the symlink. This is useful for preventing extra copies of directories already copied elsewhere in the tree.

Combining both options would allow you to follow symlinks for files and handle directory symlinks, specially to avoid unnecessary duplication. Here's how you might use rsync to synchronize an NFS-mounted directory (with symlinks) to a local directory (which you could then upload to an Amazon EFS):

rsync -avzKL /path/to/nfs/mount/ /path/to/efs/mount/

This command:

  • -a: Archive mode; equals -rlptgoD (no -H), preserving almost everything (except hard links) and ensuring that symbolic links, devices, attributes, permissions, ownerships, etc., are preserved.
  • -v: Verbose, which increases the amount of information you're given during the transfer.
  • -z: Compresses file data during the transfer.
  • -K: Ensures that symlinks to directories are treated as the directory they link to.
  • -L: Copies the files that symlinks point to rather than copying them themselves.

You can always create a local copy with all symlinks resolved,and then use DataSync to transfer this data to Amazon EFS.

If the answer is helpful, please click "Accept Answer" and upvote it.

profile picture
EXPERT
answered 3 months ago
profile picture
EXPERT
reviewed 2 months ago
profile picture
EXPERT
reviewed 2 months ago
1

You can write a script (e.g., Python or shell script) on your on-premise system that traverses the directory structure and identifies symbolic links. The script can then follow each link, determine the target file path, and copy the target file to the designated location in EFS using a separate tool like aws s3 cp (assuming you're mounting EFS as an S3 bucket). This approach requires some scripting knowledge but gives you full control over the process. Also, Investigate third-party data migration tools like AWS Migration Hub , Gs Richcopy 360, CloudFuze, or CloudEndure Migrate. These tools often offer functionalities beyond basic file copying, including following symbolic links and replicating data structures. However, they might have additional licensing costs compared to using native AWS services.

posike
answered 2 months ago
0

Thanks for answering everyone. As mentioned above, there were no ways to transfer symlink target file using datasync, so I mounted EFS then used rsync. Hope this function will be added in near future.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions