SageMaker Studio with custom EFS: permission denied for write operations

Hello, did any solution work with you because i'm currently facing the same problem?

It is a serious shortcoming of the new Studio. I've asked AWS Support and here is what I've got in return:

Hello,

Warm Greetings from AWS Support Engineering. My name is ... from SageMaker team and I'm glad to assist you with this case today.

From the case note I understood that you are using SageMaker Studio with an existing custom file system (EFS)he SageMaker user role has a policy AdministratorAccess attached. The solution works well with ReadOnly access, but getting the "Permission denied: custom-file-systems/efs/fs-..." error for the write operations.

To begin with, it is related with POSIX permissions. Please note that from the EFS side, only root user is able to directly modify the EFS root directory by default and this user has user ID(UID) and group ID(GID) of 0 [1].

On the other hand, SageMaker Studio uses the user ID of 200001 and group ID of 1001 by default [2].

Please note that currently the custom EFS permissions should be handled with the help of other services like EC2 and outside of sagemaker. We need to grant the necessary permissions to the posix user by mounting the EFS to an EC2 instance due to the configured posix user lacking permissions for the file system path.

When we try to mount / path as a "FileSystemPath" which is always owned by the root user (UID: 0 & GID:0). Only root users will be able to write data to this path. Hence, We should provide 777 permission to the EFS directory using EC2 instance.

Since sagemaker uses default value for Uid: 200001 and Gid: 1001 when we don't provide CustomPosixUserConfig parameter and also setting root permission is not allowed by sagemaker. The valid CustomPosixUserConfig parameter values are mentioned here [+] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CustomPosixUserConfig.html

If you don't want to provide 777 permission (full permission to all users) and to fine grain access using POSIX level permission, please follow the below approach:

For instance, if not already, try mounting '/test' path as a "FileSystemPath" and configure UID and GUID for one of the user profile as below: this can be done by the commands mentioned in the documentation here[3]

———————

"DomainId": "d-abc3xxxxxxx", "UserProfileArn": "arn:aws:sagemaker:eu-west-1:8111****8111:user-profile/d-wr3xxxxxxx/efs-1", Status": "InService", "CustomPosixUserConfig": { "Uid": 200020, "Gid": 1003 }, "CustomFileSystemConfigs": [ { "EFSFileSystemConfig": { "FileSystemId": "fs-000721xxxxx", "FileSystemPath": "/test" }

——————

Then we have to manage the POSIX level permission using the EC2 instance as below:

1. Start EC2 instance and mount EFS path. (Eg..In my case I mounted as /root/efs)
2. We have to create the Posix mentioned in the "CustomPosixUserConfig" in order to provide posix level permission. Steps are provided below:

[root@ip-171-1-171-197 ~]# cd /root/efs [root@ip-171-1-171-197 efs]# mkdir test [root@ip-171-1-171-197 efs]# sudo useradd -u 200020 efs-1 --->create user with uid [root@ip-171-1-171-197 efs]# sudo groupmod -g 1003 efs-1 ----> change gid [root@ip-171-1-171-197 efs]# id efs-1 ----> check the user is created with proper id's uid=200020(efs-1) gid=1003(efs-1) groups=1003(efs-1) ---> verified that uid and gid is fine. [root@ip-171-1-171-197 efs]# chown -R 200020:1003 test ---> changing ownership [root@ip-171-1-171-197 efs]# chmod -R 700 test ----> changing permission

———————

Note: 700 permission should allow access only to this user and restrict all other users even user mount with / "FileSystemPath". If you want you can change it to 755, others can read / write into it.

3. After the above changes, domain user "efs-1" can access the /test EFS path inside the Sagemaker spaces.

In summary, As of now, custom EFS permission should be handled with the help of other services like EC2 and outside of sagemaker.

Unfortunately, the only way to write on custom EFS can be achieve by the procedure that I shared above. As the custom EFS attach can only be access by root user and root access is achievable under studio hence the work around is by changing the root user permission to POSIX user and then use it via Studio to perform write action on EFS. This is know issue and our internal team is currently working on it to improve the process and reduce ad-hoc activity o