How to bring customer S3 bucket to Pcluster3.8.0 FSX/Lustre

0

I have a pre-existing S3 bucket-in-different-account (which we use daily for old 2.x PCluster fsx-lustre). We'd like to use that pre-existing S3 bucket and use it for new clusters using fsx-lustre in PCluster 3.8.0.

WHAT-DOESN"T-WORK So far I've resorted to hand-jamming permissions on random pcluster3 roles in order to give the PCluster3 ui access to the customers S3 bucket. All that gets me is PCUI sends the CloudFormation w/params for cluster creation. But, that fails in CloudFormation with...

Amazon FSx is unable to create Service-Linked-Role to access the S3 bucket. 
Ensure the IAM role or user you are using has the required permissions. 
For more details, visit https://docs.aws.amazon.com/fsx/latest/LustreGuide/setting-up.html#fsx-adding-permissions-s3. 
(Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: eb6ce2e0-6c1f-4c6f-bb7c-d71cad7ea27c; Proxy: null)

and the Stack is reverted...

WHAT DOES WORK If I abandon the customers S3 bucket (again in a different account), and I use the s3 bucket that pcluster3 creates as part of the UI thing parallelcluster-REDACTED-v1-do-not-delete I'm able to get a successfully created cluster with an fsx/ mount.
HOWEVER, I'd like to avoid copying everything over from the customer's S3 bucket to this S3 bucket.

alfreds
asked 2 months ago85 views
3 Answers
1
Accepted Answer

I made some progress, at least I got past that error message and onto the next error.

I already had cross-acount permissions working as I stated in the comments to both initial answers so that wasn't my issue.

I spelled out in detail what I had to do in my bug https://github.com/aws/aws-parallelcluster/issues/6122

basically, you have to go heavy spelunking into finding which pcluster3 role is needing the service-linked-role and add a statement to the already full (10 max) policy role.

Unfortunately, you'll then get the next productivity killer

Amazon FSx is unable to import objects from the linked data repository. Please file a ticket at https://console.aws.amazon.com/support/home#/. While filing your ticket, please include your file system ID and name of the linked data repository.

So now I'm onto that.

alfreds
answered 2 months ago
profile picture
EXPERT
reviewed a month ago
1

To integrate your pre-existing S3 bucket from a different account with PCluster 3.8.0 FSx/Lustre, you'll need to establish cross-account access and ensure the appropriate permissions are set up. Follow these steps:

  1. Set Up Cross-Account Access:

    • Create a role in your account (let's call it Account A).
    • Create another role in the target account (Account B) with a trusted policy that allows access from Account A.
    • This will enable seamless communication between the two accounts.
  2. Grant Access between Accounts:

    • Define a trusted policy in Account A to allow access from Account B.
    • Similarly, create a trusted policy in Account B to allow access from Account A.

    Trusted Policy for Account A:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-A:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }

    Trusted Policy for Account B:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-B:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }
  3. Configure Permissions for Amazon FSx:

    • Follow the guidelines provided in the Amazon FSx for Lustre documentation to set up the necessary permissions for accessing S3 buckets. You can find detailed instructions here.

Additional resources:

profile picture
EXPERT
answered 2 months ago
  • I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    
0

Hello, it seems that you would like to set up fsx cross account S3 bucket access. That will required some additional settings for cross-account bucket access. Could you check if the following link helpful? https://repost.aws/questions/QUPA2hF7XSRamoN8sXgyovBw/how-to-access-bucket-from-another-account-to-fsx

answered 2 months ago
  • COPY/PASTING my response to similar answer...

    I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions