Sagemaker Notebook Instance with VPC internet disabled unable to start training

Hello,

I have setup a VPC in the following manner:

1. one public subnet with a public route-table connected to internet gateway
2. two private subnets with a private route-table connected to NAT gateway
3. created a vpc end-point for s3 gateway
4. security group with all traffic inbound and outbound

I created a notebook instance, attached the above vpc in network settings, mentioned security-group and disabled internet.

I created a docker image with training scripts and pushed it to ecr, data is on s3 bucket.

I created a sagemaker estimator with subnets arguments including private subnet-ids and security_id, the image-uri and s3 bucket information.

When I perform estimator.fit the notebook is stuck without any error or warnings, what could be the issue ?