Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
How to Configure Custom DNS Resolution for SageMaker Processing Jobs Using Private VPC Endpoints and Route 53 Resolver?
I'm running AWS SageMaker Processing jobs inside a VPC, and I want these jobs to resolve internal service names (e.g., internal.api.corp) using custom DNS rules. My VPC is configured with Route 53 Resolver inbound/outbound endpoints and forwarding rules to on-premises DNS servers for specific domains.
However, I'm noticing that SageMaker Processing jobs sometimes fail to resolve these internal domains, even though EC2 instances in the same subnets can resolve them correctly. I suspect there are differences in how SageMaker jobs interact with VPC DNS settings, especially with Route 53 Resolver rules and conditional forwarding.
How can I ensure that SageMaker Processing jobs consistently use my custom DNS resolution setup? Are there any limitations or caveats with SageMaker Processing jobs and Route 53 Resolver forwarding rules? Do I need to set any special configurations (e.g., DNS options on the VPC, specific IAM permissions, or SageMaker job parameters) to make this work? Is there any AWS documentation or best practices for troubleshooting DNS issues specifically for SageMaker Processing jobs in VPCs with custom DNS setups? Any insights or experiences would be greatly appreciated, as I haven't found detailed guidance in the AWS documentation or forums.
- Language
- English
- Newest
- Most votes
- Most comments
VPC dns Settings Make sure the VPC attached to the SageMaker job has: Enable DNS Resolution: Yes Enable DNS Hostnames: Yes
You can check these under VPC -->Your VPC--->Actions --->Edit DNS resolution/hostnames
Ensure your SageMaker job is launched into the private subnet that: Has access to the Route 53 Resolver outbound endpoint Has security group that allows outbound traffic to DNS resolvers (udp port 53,)
Verify this outbound endpoint is in the same VPC or shared via AWS RAM Forwarding rules are set for domains like internal.api.corp and point to the on-prem DNS IP Use Route 53 Resolver logging to confirm whether queries from SageMaker jobs are reaching the Resolver endpoint
Workeraroud launch your SageMaker Processing job with a custom script to overide resolv.conf you can use --network--config in your SageMaker SDK or API calls to specify networking
Ensure the SageMaker execution role has access to the VPC configuration
You can also use VPC endpoints if your internal services are hosted in AWS, consider exposing them via VPC interface Endpoints and routing through the AWS VPC DNS namespace
Relevant content
- asked a year ago
- asked 2 years ago
