I want to troubleshoot issues when I access an Amazon SageMaker Project in SageMaker Studio.
Resolution
SageMaker Project lets you manage and orchestrate end-to-end ML solutions. When you create or access projects from the Studio environment, you might have issues due to one or more of the following reasons:
- Missing permissions
- Network issues in VPC only mode
- Studio user interface issues
Missing permissions
SageMaker Projects rely on AWS Service Catalog to use or create project templates and provision AWS resources for your accounts. Your Studio domain users can access or view these templates only when you grant Projects permissions.
If your user doesn't have Projects permissions, then your user gets an error message similar to the following one:
Amazon SageMaker project templates aren't enabled for your account.
Contact your administrator to enable SageMaker project templates.
To resolve this issue, be sure to grant Projects permissions for the administrator and domain execution role users.
Also, you might have permission issues because one of the following items is missing, deleted, modified, or recreated manually:
- A service roles that was created when you activated SageMaker Project
- Amazon SageMaker JumpStart
To resolve this issue, complete the following steps to turn off SageMaker project templates and JumpStart:
1. Open the SageMaker console.
2. From the navigation pane, choose Domains.
3. Use the button next to the domain name to select your domain. Then, at the top of the Domains panel, choose Edit.
4. On the General settings page, choose Next.
5. Under SageMaker Projects and JumpStart, turn off the Enable Amazon SageMaker project templates and Amazon SageMaker JumpStart for this account. Also, turn off Enable Amazon SageMaker project templates and Amazon SageMaker JumpStart for Studio users switches.
6. Choose Next.
7. On RStudio settings page, choose Next.
8. On Amazon SageMaker Canvas settings page, choose Submit.
9. Delete all the roles that start with AmazonSageMakerServiceCatalogProduct*.
10. Use the preceding instructions to turn on the toggle switches that activate SageMaker project templates and JumpStart. This step creates several new roles for you. To view the list of your roles: Under Projects, choose the Apps tab, and then choose Control Panel.
Note: To have the required permissions to access the AWS Service Catalog console, attach the managed policy SageMakerFullAccess to your Studio users' execution roles.
Network issues in VPC only mode
Internet access isn't active with the VPC only network access type. Therefore, you can't run a Studio notebook unless the following conditions are true:
- Your VPC has an interface endpoint to the SageMaker API and runtime or a NAT gateway with internet access.
- Your security groups allow outbound connections.
If you don't need NAT gateway access for your SageMaker project, then you must create interface VPC endpoints (AWS PrivateLink) to connect to com.amazonaws.[region].servicecatalog. For example, if you're using Studio in the eu-west-1 AWS Region, then use com.amazonaws.eu-west-1.servicecatalog. For more information, see Requirements to use VPC only mode.
To verify that your Studio's network settings allow connection to SageMaker API and Service Catalog endpoints, run the following commands from a Studio system terminal:
Check access to SageMaker API endpoint:
curl -v https://api.sagemaker.example-region.amazonaws.com
Check access to the Service Catalog endpoint in the desired Region:
curl -v https://api.sagemaker.example-region.amazonaws.com
If you get the Connection timed out error when running these commands, then verify that your VPC network settings are configured for VPC only mode.
View the list of available projects from the system terminal by running the following command:
$ aws sagemaker list-projects --sort-by CreationTime --sort-order Descending
If you can't view the same list in the SageMaker Studio interface, then make sure to update your Studio's domain. Then, do the following:
1. Launch SageMaker Studio.
2. In the left sidebar, choose Commands.
3. Search for Reset Application State in the Search bar, and then choose this option.
Note: In JupyterLab 3, there's no Command Palette on the left sidebar by default. To add the Command Palette: On the menu, choose View, and then on the menu bar, choose Activate Command Palette. Or, you can use the keyboard shortcut command Ctrl + Shift + C.