How do I troubleshoot network connectivity issues in my Batch jobs?

Short description

AWS Batch jobs that run in private subnets require network connectivity to AWS service endpoints. If you incorrectly configured your network connection or it's disrupted, then jobs fail with timeout errors, image pull failures, or credential retrieval issues. These errors occur even when you correctly configure AWS Identity and Access Management (IAM) permissions and compute resources.

Resolution

Important: The following troubleshooting steps resolve network connectivity and timeout issues for Batch jobs that run in Amazon Elastic Container Service (Amazon ECS) on Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate. For general Batch troubleshooting instructions, see Troubleshooting AWS Batch.

Check your VPC endpoint configuration for private subnets

For Batch compute environments that use private subnets without a NAT gateway, you must configure virtual private cloud (VPC) endpoints to allow jobs to communicate with AWS services. For more information about VPC endpoint requirements for AWS Batch, see Considerations for AWS Batch.

To check the VPC endpoint configuration for private subnets, complete the following steps:

1. Open the Amazon Virtual Private Cloud (Amazon VPC) console.
2. In the navigation pane, choose Endpoints.
3. Check that the following necessary endpoints exist in the VPC that your Batch compute environment uses:
   com.amazonaws.example-region.ecr.api (Interface)
   com.amazonaws.example-region.ecr.dkr (Interface)
   com.amazonaws.example-region.s3 (Gateway)
   com.amazonaws.example-region.sts (Interface)
   com.amazonaws.example-region.logs (Interface)
   If your Batch job runs on Amazon EC2, then also check that the following necessary endpoints exist:
   com.amazonaws.example-region.ecs-agent (Interface)
   com.amazonaws.example-region.ecs-telemetry (Interface)
   com.amazonaws.example-region.ecs (Interface)
   Note: Replace example-region with your AWS Region.
4. If any of the preceding endpoints don't exist, then choose Create endpoint, and then create the required endpoints in your compute environment's VPC.
5. For each interface endpoint, check that the security group allows inbound HTTPS traffic on port 443 from the compute environment's security group.
6. Check that the security group that's attached to your Batch compute environment allows outbound traffic on port 443.
7. If you restricted your outbound rules to specific destinations, then check that they include the VPC endpoint security groups or your VPC's CIDR range.

Important: If your jobs access other AWS services, then also create VPC endpoints for the other services.

Check the VPC endpoint policies

Restrictive VPC endpoint policies block access to AWS services, even when your IAM role has the required permissions.

To check your VPC endpoint policies, complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Endpoints.
3. Select each endpoint that your Batch environment uses, and then choose the Policy tab.
4. Check that the policy allows the actions that your Batch jobs require.

For the Amazon Simple Storage Service (Amazon S3) gateway endpoint, check that the policy contains GetObject permissions for the Regional Amazon Elastic Container Registry (Amazon ECR) layer bucket. If the policy doesn't contain GetObject permissions, then add the following statement to your policy:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::prod-example-region-starport-layer-bucket/*"]
  }]
}

Note: Replace example-region with your Region.

Check that the policy allows the actions that your execution role and job role require for each interface endpoint. For example, if your job uses AWS Key Management Service (AWS KMS) decryption, then the AWS KMS VPC endpoint policy must allow the kms:Decrypt action for your role.

If you continue to receive errors, then temporarily set the policy to Full Access, and then test your job. If the job succeeds, then include only the minimum required permissions in the policy.

Resolve container image pull failures

If your Batch jobs fail with "CannotPullContainerError or manifest read errors," then the compute environment can't download the container image from Amazon ECR. For more information, see Amazon ECR interface VPC endpoints (AWS PrivateLink).

To resolve container image pull failures, complete the following steps:

1. Open the AWS Batch console.
2. In the navigation pane, choose Job definitions, and then select your job definition.
3. Check that the Execution role field contains a valid IAM role Amazon Resource Name (ARN).
4. Check that the execution role has the following permissions:
   ecr:GetAuthorizationToken
   ecr:BatchCheckLayerAvailability
   ecr:GetDownloadUrlForLayer
   ecr:BatchGetImage
   For more information about required permissions, see Using Amazon ECR images with Amazon ECS.
5. For images in a different AWS account, check that the source account's repository policy grants your role access to the required Amazon ECR actions.
6. Check that the following network connectivity paths correctly function:
   ECR API endpoint for authorization tokens
   ECR .dkr endpoint for image manifests
   S3 gateway endpoint for image layers

Important: The execution role (executionRoleArn) and the job role (jobRoleArn) aren't interchangeable.

If Batch pulls large images that intermittently time out, then use multi-stage Docker builds to reduce your image size. For public base images, use Amazon ECR pull-through caches, instead.

Resolve credential retrieval failures

If your Batch jobs intermittently fail with "Unable to retrieve credentials" or "AccessDeniedException" errors, then complete the following steps:

1. Open the AWS Batch console.
2. In the navigation pane, choose Job definitions, and then select your job definition.
3. Check that the Job role field contains a valid IAM role ARN. For more information, see Container properties.
4. If your compute environment uses a custom launch template, then check that you set the HttpPutResponseHopLimit metadata option to 2.
5. For Java applications that use AWS SDK v2, set the asyncCredentialUpdateEnabled value to true for the ContainerCredentialsProvider.
6. To configure a retry strategy to your job definition to handle transient credential failures, take the following actions:
   Set the Attempts value to 3 or higher.
   Add evaluateOnExit rules for the exit codes that your application returns on credential failures. 
   For statusReason patterns, such as "Host EC2*", add evaluateOnExit rules to retry on infrastructure issues.
7. For compute environments that use encrypted Amazon Elastic Block Store (Amazon EBS) volumes with a customer managed key, add the following permissions to the KMS key policy for the AWSServiceRoleForAutoScaling role:
   kms:GenerateDataKeyWithoutPlaintext
   kms:CreateGrant.
   Note: By default, Amazon Linux 2023 (AL2023) enforces IMDSv2 with the HttpPutResponseHopLimit value set to 1. If you migrate from Amazon Linux 2 (AL2) and receive credential errors, then check that your launch template sets the HttpPutResponseHopLimit value to 2.

Check the subnet IP address capacity

If an error message contains "InsufficientFreeAddressesInSubnet" or elastic network interface creation failures, then your subnets have no available IP addresses. Your compute environment then fails to launch new instances.

To resolve subnet IP address capacity, complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Subnets.
3. Select each subnet that your Batch compute environment uses.
4. On the Details tab, under Available IPv4 addresses, check the number of available IP addresses.

If the number of available IP addresses is low, then add subnets to your compute environment configuration in different Availability Zones. For high-concurrency Batch workloads, use subnets with 20 or more CIDR blocks.

For Batch compute environments that are based in Amazon Elastic Kubernetes Service (Amazon EKS), activate Amazon VPC container network interface prefix delegation.

Note: Each Amazon EC2 instance uses one IP address for each network interface. If you use awsvpc networking mode, then each task also uses a separate IP address from the subnet.

Check for AWS infrastructure events

If your Batch jobs fail with timeout errors and you didn't change your configuration, then AWS infrastructure events affect connectivity in your Region.

To check for active AWS infrastructure events in the Region where your Batch compute environment runs, complete the following steps:

1. Open the AWS Health Dashboard.
2. Check whether same-Region operations succeed when cross-Region calls time out so that you don't experience connectivity issues between Regions.
3. Check whether the event affects multiple unrelated workloads during the same time window.
4. If the issue correlates with an active AWS Health event, then open an AWS Support case. Include the affected timestamp window (in UTC), the source and destination Regions, and Health Dashboard Event IDs.
   Note: Service-side issues cause terminations that don't match Auto Scaling activity or Spot interruption notices for EC2 instances in your Batch compute environment. To resolve this issue, open a Support case with the terminated instance IDs and termination timestamps in UTC.

Related information

AWS Batch execution IAM role

Jobs stuck in a RUNNABLE status