EC2 + IPv6-Only + Docker + awslogs driver to CloudWatch (not working)

TL;DR: Docker awslogs driver failing with "i/o timeout" to CloudWatch Logs over IPv6 on EC2

Problem Description:

We are encountering a persistent "i/o timeout" error when our Docker container, running on an EC2 instance with public IPv6-only connectivity (no public IPv4), attempts to send logs to CloudWatch Logs using the awslogs driver. The error message consistently shows an attempt to dial tcp <IPv4_ADDRESS>:443: i/o timeout, indicating the awslogs driver is still trying to connect via IPv4 despite the environment being configured for IPv6. We want to avoid using the Cloudwatch VPC Endpoint with Private IPv4 because of the extra VPC Endpoint fees.

Environment Details:

• EC2 Instance: Amazon Linux 2, configured with only private IPv4 and public IPv6 addresses. No public IPv4 is assigned.
• Docker Version: Whatever latest version that comes with Amazon Linux
• AWS Region: us-west-2
• CloudWatch Logs: Public endpoint for us-west-2 is logs.us-west-2.amazonaws.com. This service is confirmed to support IPv6.

Steps Taken and Observations:

1. Initial Setup: The webapp-service Docker container is configured to use the awslogs driver.
2. VPC Endpoint Removal: We initially attempted to use a CloudWatch Logs VPC endpoint, but due to cost considerations, this was removed. The current goal is to utilize the EC2 instance's public IPv6 connectivity to reach the public CloudWatch Logs service endpoint.
3. Docker Daemon IPv6 Configuration:
   • We ensured /etc/docker/daemon.json on the EC2 instance is configured as follows:

     {
       "ipv6": true,
       "fixed-cidr-v6": "fd00::/80"
     }

   • This configuration is verified to be correctly applied on the remote host.
   • An aggressive Docker daemon reset (disable, stop, remove all networks and containers, then re-enable and start) is performed to ensure a clean state and proper application of daemon.json changes.
   • The Docker daemon is confirmed to be active after these steps.
4. docker run Command: The container is launched with the following command (simplified for clarity):

   /usr/bin/docker run --name webapp-service \
     -p 8080:8080 \
     -e ENVIRONMENT_NAME=myapp-dev \
     -e AWS_REGION=us-west-2 \
     -e AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE=IPv6 \
     --log-driver=awslogs \
     --log-opt awslogs-group=/ec2/myapp-dev-webapp-service-service-v1 \
     --log-opt awslogs-region=us-west-2 \
     webapp-service:latest

   • We explicitly added -e AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE=IPv6 to encourage the AWS SDK (used by awslogs) to prefer IPv6.
5. Error: Despite all these configurations, the docker run command fails with:

   docker: Error response from daemon: failed to create task for container: failed to initialize logging driver: failed to create Cloudwatch log stream: operation error CloudWatch Logs: CreateLogStream, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post "https://logs.us-west-2.amazonaws.com/": dial tcp 44.234.123.82:443: i/o timeout.
   

   The dial tcp <IPv4_ADDRESS>:443 clearly shows an IPv4 connection attempt, which times out because the EC2 instance lacks public IPv4.

Question:

How can we force the Docker awslogs driver (or the underlying AWS SDK it uses) to resolve the logs.us-west-2.amazonaws.com endpoint to its IPv6 address and use IPv6 for communication, given that the Docker host has public IPv6 connectivity and the Docker daemon is configured for IPv6?

Are there any specific awslogs driver options, Docker network configurations, or AWS SDK environment variables (beyond AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE=IPv6) that need to be set to ensure IPv6 preference for CloudWatch Logs communication from a Docker container on an IPv6-only public EC2 instance?

Any guidance on how to diagnose why the awslogs driver is still attempting IPv4 connections would be greatly appreciated.