EC2 Instance in Dual-Stack VPC Cannot Reach S3 via Gateway Endpoint

Hello,

I am encountering a persistent networking issue where an EC2 instance, launched via CloudFormation in a public subnet of a dual-stack VPC, with Private IPv4 and public-routable IPv6 but no Public IPv4, is unable to establish connectivity to Amazon S3 via a configured S3 VPC Gateway Endpoint.

A ping test to s3.us-west-2.amazonaws.com from the EC2 instance consistently shows 100% packet loss. This prevents aws s3 cli from working.

Configuration Details:

• VPC Type: Dual-stack.
• EC2 Instance: Launched in a public subnet (IPv4 and IPv6 enabled without a public IPv4).
• S3 VPC Gateway Endpoint: Configured within the same VPC and associated with the public subnet's route table.
• IAM Role for EC2: Has necessary s3:GetObject and s3:ListBucket permissions for the target S3 bucket.
• S3 Bucket Policy: No explicit denying policies are present on the target S3 bucket.
• EC2 Security Group: Egress rules allow all outbound IPv6 traffic (::/0).
• UserData Script: Includes a ping test to s3.<us-west-2>.amazonaws.com which fails.

Relevant CloudFormation Template Snippets:

(NOTE: template has been de-identified and summarized by AI and may contain errors not present in the actual attempted deployment)

  RendezvousEc2Instance:
    Type: AWS::EC2::Instance
    Properties:
      # ... other properties ...
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - !GetAtt RendezvousEc2SecurityGroup.GroupId
          SubnetId: !ImportValue
            Fn::Sub: "CorePublicSubnet1Id-${EnvironmentName}" # Using a public subnet for IPv6 internet access
          Ipv6AddressCount: 1
          AssociatePublicIpAddress: false # Explicitly set to false for IPv6-only instance.
      # ... other properties ...

  S3VpcEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !ImportValue
        Fn::Sub: "CoreVpcId-${EnvironmentName}"
      ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3"
      RouteTableIds:
        - !ImportValue
          Fn::Sub: "CorePublicSubnet1RouteTableId-${EnvironmentName}"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal: "*"
            Action:
              - "s3:GetObject"
              - "s3:ListBucket"
            Resource:
              - !Sub 'arn:aws:s3:::${ArtifactS3Bucket}'
              - !Sub 'arn:aws:s3:::${ArtifactS3Bucket}/*'

  RendezvousEc2LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      # ... other properties ...
      UserData: !Base64
          Fn::Sub:
            - |
              #!/bin/bash -xe
              set -e
              exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
              echo "Starting UserData script at $(date)"
              # ... Docker installation ...
              echo "Testing connectivity to S3 endpoint..."
              if ! ping -c 3 s3.us-west-2.amazonaws.com; then
                echo "ERROR: Failed to ping S3 endpoint. Network connectivity issue. Exiting UserData."
                exit 1
              fi
              echo "Connectivity to S3 endpoint successful."
              echo "Syncing Docker registry data from S3..."
              if ! aws s3 sync s3://${ArtifactS3Bucket}/registry-data /opt/registry-data; then
                echo "ERROR: Failed to sync Docker registry data from S3. Exiting UserData."
                exit 1
              fi
              echo "Registry data synced."
              # ... rest of script ...
            - LambdaFunctionUrl: !GetAtt RendezvousCfnSignalLambdaFunctionUrl.FunctionUrl

Observed Behavior:

• EC2 instance fails to ping s3.us-west-2.amazonaws.com with 100% packet loss.
• A critical aws s3 sync command within the UserData script hangs, leading to script termination.
• CloudFormation stack deployments consistently fail due to the UserData script exiting prematurely.

Expected Behavior: The EC2 instance should be able to successfully communicate with Amazon S3 via the S3 VPC Gateway Endpoint, allowing the aws s3 sync command to complete.

Troubleshooting Steps Taken:

1. IAM Permissions: Verified that the EC2 instance's IAM role has s3:GetObject and s3:ListBucket permissions.
2. S3 Bucket Policy: Confirmed that the target S3 bucket has no explicit denying policies.
3. EC2 Security Group: Verified that the EC2 instance's security group has an egress rule allowing all IPv6 outbound traffic.
4. S3 VPC Endpoint Configuration:
   • The S3 VPC Endpoint resource is defined in the CloudFormation template.
   • It is configured to associate with the public subnet's route table.
   • AWS CLI commands confirm the S3 VPC Endpoint is in CREATE_COMPLETE state.
   • However, inspecting the associated route table shows no automatically injected route for the S3 Prefix List pointing to the VPC Endpoint.
5. Manual Route Creation Attempt: Attempted to manually create the missing route using aws ec2 create-route, but it failed with InvalidParameterValue: Cannot create or replace a prefix list route targeting a VPC Endpoint., indicating that the endpoint is expected to manage its own routes.
6. UserData Script Enhancements: Added a ping test and explicit error handling to the UserData script, confirming the network issue.

The core issue appears to be that the S3 VPC Gateway Endpoint is not correctly injecting its routes into the associated route table, despite being configured to do so within the CloudFormation template. This prevents any traffic from the EC2 instance from reaching S3 privately.

The end-goal is to be able to access S3 from this EC2 instance without the addition of a NAT, Public IPv4, or any other cost-additive components.

Is there a solution?