CloudFront 504 Gateway Timeout for EC2 VPC Origin (IPv6 + Private IPv4)

Description:

We are attempting to remove Public IPv4, NAT, and ALB overhead costs from our infrastructure, one use-case at a time. The current use-case is web-facing stateful services (not fit for Lambda) that do not require high availability (so for example, load-balancing or blue/green deployments are excluded) and also are not containerized (so ECS/ECR are excluded from the solution). The short-term objective for technology demonstration purposes, is to use CloudFormation to stand up a stand-alone EC2 instance serving web traffic to the Internet.

We have created this simple template with an Ngix default page to demonstrate the issue:

We are experiencing a persistent "504 Gateway Timeout" error from a CloudFront distribution configured with an EC2 VPC Origin. The origin points to an EC2 instance (i-03fb7666cc52c9636) running Amazon Linux 2023, which is intended to be IPv6-only with a private IPv4 address.

Current Setup:

• EC2 Instance:
  • Launched in a public subnet with a single IPv6 address and AssociatePublicIpAddress: false (no public IPv4).
  • UserData script successfully installs and starts Nginx, serving a default "Hello World" page on port 80.
  • System logs confirm Nginx is running and IPv6 outbound connectivity is working.
• Security Group (RendezvousEc2SecurityGroup):
  • Ingress rule for TCP port 80 from ::/0 (public IPv6).
  • Diagnostic change applied: Added a temporary ingress rule for TCP port 80 from 10.0.0.0/8 (private IPv4 range) to allow CloudFront's VPC origin to connect. This did not resolve the 504 error.
• CloudFront Distribution (RendezvousCloudFrontDistribution):
  • Configured with IPV6Enabled: true.
  • Origin uses VpcOriginConfig referencing RendezvousVpcOrigin.
  • DomainName for the origin is set to !GetAtt RendezvousEc2Instance.PrivateDnsName.
  • HTTPPort is 80, and OriginProtocolPolicy is http-only.

Problem: Despite Nginx running on the EC2 instance and security groups allowing traffic on port 80 (both IPv6 public and a broad IPv4 private range for diagnosis), CloudFront consistently returns a 504 Gateway Timeout. This indicates CloudFront is unable to establish a connection with the EC2 instance.

Request: What could be causing this?

Here are some relevant bits of the template:

  RendezvousEc2LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateName: !Sub "${EnvironmentName}-RendezvousEc2LaunchTemplate"
      LaunchTemplateData:
        ImageId: !Ref AmiId # Dynamically provided AMI ID
        InstanceType: !Ref InstanceType
        IamInstanceProfile:
          Arn: !GetAtt RendezvousEc2InstanceProfile.Arn
        KeyName: !Ref KeyName # WHY: Enables SSH access to the EC2 instance using the specified key pair.
        UserData: !Base64
          Fn::Sub:
            - |
              #!/bin/bash -xe
              set -e
              echo "Starting UserData script at $(date)"
              exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
              echo "Starting UserData script at $(date)"

              # Install necessary packages for Amazon Linux 2023
              # WHY: dnf is the package manager for AL2023. We need aws-cli and nginx.
              # MISTAKES TO AVOID:
              # 1. Missing aws-cli: The signaling script will fail without it.
              # 2. Using yum/amazon-linux-extras: These are for AL2, not AL2023.
              dnf update -y
              dnf install -y nginx bind-utils curl aws-cli --allowerasing

              echo "Nginx, curl, and aws-cli installed."

              echo "Starting IPv6 connectivity diagnostics at $(date)"
              echo "Configuring IPv6 DNS resolver..."
              echo "nameserver 2001:4860:4860::8888" > /etc/resolv.conf
              echo "Checking /etc/resolv.conf after modification:"
              cat /etc/resolv.conf
              echo "Testing general outbound IPv6 connectivity to ipv6.google.com:"
              curl -6 -v https://ipv6.google.com
              echo "Finished IPv6 connectivity diagnostics at $(date)"

              # Configure Nginx for "Hello World"
              echo "Configuring Nginx..."
              echo "<h1>Hello from Rendezvous EC2 (IPv6-only)</h1>" > /usr/share/nginx/html/index.html
              systemctl start nginx
              systemctl enable nginx
              echo "Nginx started and enabled."

              # Signal CloudFormation that the instance is up
              # WHY: The cfn-signal utility cannot be used on an IPv6-only instance because
              # the CloudFormation API endpoint is IPv4. Instead, we invoke a Lambda function
              # which can then signal on our behalf.
              # MISTAKES TO AVOID:
              # 1. Missing IAM Permissions: Ensure the EC2 instance role has 'lambda:InvokeFunctionUrl' permission.
              # 2. Incorrect Payload: The payload must match what the Lambda function expects.
              # 3. Script Errors: Any error before this point will prevent the signal from being sent,
              #    causing the CloudFormation stack to time out and roll back, which is the correct behavior.
              echo "Signaling CloudFormation via Lambda Function URL..."

              # MISTAKES TO AVOID:
              # 1. Fragile Metadata Service: Relying on the metadata service can be fragile.
              #    Using a known tag and passing the region directly is more robust.
              # 2. Incorrect Function URL: Ensure the Function URL is correctly retrieved and used.
              # 3. Unescaped CloudFormation Variables: CloudFormation's !Sub function will try to
              #    resolve bash variables (e.g., $SIGNAL_PAYLOAD) as CloudFormation
              #    parameters or pseudo-parameters, leading to deployment failures.
              #    By directly embedding !Sub values and simplifying dynamic parts, we avoid this.
                  SIGNAL_PAYLOAD='{ "LogicalResourceId": "RendezvousEc2Instance", "Status": "SUCCESS" }'
                  LAMBDA_FUNCTION_URL="${LambdaFunctionUrl}"

                  curl -6 -X POST -H "Content-Type: application/json" \
                    -d "$SIGNAL_PAYLOAD" \
                    "$LAMBDA_FUNCTION_URL"

                  echo "UserData script finished at $(date). Lambda Function URL invoked."
            - LambdaFunctionUrl: !GetAtt RendezvousCfnSignalLambdaFunctionUrl.FunctionUrl
        InstanceMarketOptions:
          MarketType: spot
        TagSpecifications: # Tags for the instance launched by the Launch Template
          - ResourceType: instance
            Tags:
              - Key: Name
                Value: !Sub "${EnvironmentName}-RendezvousEc2Instance"

  # EC2 Spot Instance
  # WHY: This resource now references the Launch Template for its configuration,
  # correctly separating the instance definition from the market options.
  # It is configured with IPv6 and private IPv4, explicitly disabling public IPv4
  # to avoid associated charges.
  # MISTAKES TO AVOID:
  # 1. Missing Launch Template Reference: Ensure the LaunchTemplate property
  #    is correctly configured to point to the created Launch Template.
  # 2. Redundant Properties: Avoid duplicating properties already defined in
  #    the Launch Template (e.g., ImageId, InstanceType, UserData).
  RendezvousEc2Instance:
    Type: AWS::EC2::Instance
    Properties:
      LaunchTemplate:
        LaunchTemplateId: !Ref RendezvousEc2LaunchTemplate
        Version: !GetAtt RendezvousEc2LaunchTemplate.LatestVersionNumber
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - !GetAtt RendezvousEc2SecurityGroup.GroupId
          SubnetId: !ImportValue
            Fn::Sub: "CorePublicSubnet1Id-${EnvironmentName}" # Using a public subnet for IPv6 internet access
          Ipv6AddressCount: 1 # Request a single IPv6 address
          # MISTAKES TO AVOID:
          # 1. Assigning Public IPv4: This instance is intended to be IPv6-only.
          #    Removing AssociatePublicIpAddress ensures no public IPv4 is assigned.
          AssociatePublicIpAddress: false # WHY: Explicitly set to false for IPv6-only instance.
      Tags:
        - Key: Name
          Value: !Sub "${EnvironmentName}-RendezvousEc2Instance"
    CreationPolicy: # WHY: CloudFormation waits for a signal from the custom resource before marking the instance creation as complete.
      ResourceSignal:
        Timeout: PT10M # MISTAKES TO AVOID: Ensure this timeout is sufficient for the instance to boot and the custom resource to signal.
        Count: 1 # Expect one signal from the custom resource