DataSync symlink target file from NFS server to EFS

0

I am trying to use DataSync to copy my file from on-premise(NFS 3) to EFS. It says symbolic links are preserved when copying from NFS file server and EFS,(https://docs.aws.amazon.com/datasync/latest/userguide/special-files-copied.html#special-files-copied-symbolic-links). But I want the DataSync to copy the target file of the symlink to the EFS. I've looked into options or datasync cli, but couldn't find a solution.

Any options to suggest?(I have to keep the symlink, but target file destinations are spread out, which makes it hard to get a pattern of them)

已提问 3 个月前169 查看次数
3 回答
3
已接受的回答

If you need to actually follow the symbolic link and copy the data, you can always use rsync with specific options to handle symlinks effectively if AWS DataSync does not meet your requirements.

The -L and -K options in rsync control how symlinks are handled during the synchronization process:

  • -L or --copy-links: This option tells rsync to transform the symlinks into referent files/dirs. In other words, if you have a symlink that points to a file or directory, it will copy the actual file or directory that the symlink points to insteaf of copying the symlink itself.

  • -K or --keep-dirlinks: This option is used with -L to avoid duplicating the referent of a symlinked directory inside a copied tree again at the location of the symlink. This is useful for preventing extra copies of directories already copied elsewhere in the tree.

Combining both options would allow you to follow symlinks for files and handle directory symlinks, specially to avoid unnecessary duplication. Here's how you might use rsync to synchronize an NFS-mounted directory (with symlinks) to a local directory (which you could then upload to an Amazon EFS):

rsync -avzKL /path/to/nfs/mount/ /path/to/efs/mount/

This command:

  • -a: Archive mode; equals -rlptgoD (no -H), preserving almost everything (except hard links) and ensuring that symbolic links, devices, attributes, permissions, ownerships, etc., are preserved.
  • -v: Verbose, which increases the amount of information you're given during the transfer.
  • -z: Compresses file data during the transfer.
  • -K: Ensures that symlinks to directories are treated as the directory they link to.
  • -L: Copies the files that symlinks point to rather than copying them themselves.

You can always create a local copy with all symlinks resolved,and then use DataSync to transfer this data to Amazon EFS.

If the answer is helpful, please click "Accept Answer" and upvote it.

profile picture
专家
已回答 3 个月前
profile picture
专家
已审核 1 个月前
profile picture
专家
已审核 2 个月前
1

You can write a script (e.g., Python or shell script) on your on-premise system that traverses the directory structure and identifies symbolic links. The script can then follow each link, determine the target file path, and copy the target file to the designated location in EFS using a separate tool like aws s3 cp (assuming you're mounting EFS as an S3 bucket). This approach requires some scripting knowledge but gives you full control over the process. Also, Investigate third-party data migration tools like AWS Migration Hub , Gs Richcopy 360, CloudFuze, or CloudEndure Migrate. These tools often offer functionalities beyond basic file copying, including following symbolic links and replicating data structures. However, they might have additional licensing costs compared to using native AWS services.

posike
已回答 2 个月前
0

Thanks for answering everyone. As mentioned above, there were no ways to transfer symlink target file using datasync, so I mounted EFS then used rsync. Hope this function will be added in near future.

已回答 2 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则