My question is ultimately a generic one about how "best" to allow SageMaker resources (such as Studio instances) access to private services running in our AWS VPCs.
Specifically, I have a dataset version manager (lakefs) running on an ec2 instance in a private subnet within a VPC. Currently all traffic to this server is handled via an internet facing application load balancer so that my team can access the lakefs server's web UI from their browser, provided they are on our corporate VPN. Most of my colleagues prefer to work within SageMaker Studio (new) instances to run experiments and train models. Ultimately I would like to allow the Studio instances to authenticate with the lakefs server, so that we can fetch datasets using lakefs URIs.
To experiment, I have created a VPCOnly SageMaker domain attached to the same VPC as the lakefs server. A security group is associated to the SageMaker domain, and appears to be attached to the elastic network interfaces in our
VPC's subnets that are provisioned when a Studio Instance is created by a user. I've added this SageMaker security group to the inbound rules of my ALB's security group, and thus according to the AWS VPC Reachability Analyzer, the lakefs server is reachable from these ENIs. However, I cannot actually curl the lakefs server from within the Studio instance JupyterLab command line widget. I'm guessing this has to do with how security group sources resolve.
The public IP of the Studio instance (found via curl ifconfig.me) is just the elastic IP of the VPC subnet's NAT, more or less as expected. I could add the elastic IPs for each subnet NAT in our ALB security group inbound rules, but this strikes me as too broad and prone to security issues if the NAT is deleted and the IP re-assigned. I'd prefer to use security groups or some other approach completely that bypasses the need for direct internet connections.
AWS OFFICIALUpdated a year ago
