cloudfront cache behavior causes 504?

Hello,

A 504 error in CloudFront typically indicates that the edge location couldn't contact your origin server. Setting all TTL values to 0 forces CloudFront to always forward requests to your origin server, potentially exposing any availability or response time issues with your origin server. For detailed troubleshooting steps, refer to the AWS documentation on troubleshooting 504 errors. https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html

https://repost.aws/knowledge-center/cloudfront-troubleshoot-504-errors

Hello

Check these steps to resolve the issue

Investigate CloudFront Logs: Look for more details in CloudFront logs beyond the "originCommError" message. You might find specific error codes or messages that provide clues about the issue at the HKG1-P2 location.

Test Origin Server Response Time: Use a tool like ping or traceroute to measure the response time from the HKG1-P2 edge location to your origin server. This can help identify any network latency issues.

Increase Origin Response Timeout (if applicable): You can try increasing the CloudFront origin response timeout value for the specific cache behavior. This gives your origin server more time to respond to requests. Be aware that this is a temporary workaround and not a solution.

Check Security Groups: Ensure security groups for your origin server (or ELB) allow inbound traffic from CloudFront on the appropriate ports (typically 80 or 443).

Monitor Origin Server Health: If you're using an ELB, monitor its health metrics to identify any unhealthy instances that might be causing slow responses.

Hello,

Please fellow bellow steps.

1. Verify origin server health.

2. Review and analyze CloudFront logs.

3. Adjust cache behavior settings to balance load and performance.

4. Increase the origin response timeout.

5. Test connectivity from edge locations.

6. Verify DNS and network configurations.

7. Contact AWS support if needed.

HI Wayne

=>Here's a concise response to your CloudFront cache behavior and 504 error issue:

=>A TTL of 0 shouldn't directly cause the error. Focus on these: =>Origin Server: Ensure it's reachable (public internet) and responsive (check logs for overload or errors). Cache Behavior: No custom error page for 504 (CloudFront might keep fetching it). Adjust origin read timeout if needed.

=>For further details, refer to the official CloudFront documentation on HTTP status codes: [CloudFront status codes ON Amazon.com docs.aws.amazon.com]

Hi wayne,

please go through the below steps i hope it will helps to resolve your issue.

Steps to Troubleshoot CloudFront 504 Errors

Check Origin Server Performance:

• Latency and Response Time: Ensure your origin server (behind the ELB) can handle requests quickly and consistently. Use monitoring tools to check for spikes in latency or slow response times.
• Health Checks: Verify that ELB health checks are passing and that your origin servers are healthy.

Review CloudFront Distribution Settings:

• TTL Settings: Setting all TTLs (min, max, default) to 0 means every request will be forwarded to the origin, which can increase load and potentially cause timeouts if the origin can't handle the traffic.
• Origin Response Timeout: Increase the Origin Response Timeout in your CloudFront distribution settings to allow more time for your origin to respond.

Check Networking and Connectivity:

• Cross-region Latency: Some regions may experience higher latency when communicating with your origin server. Ensure your origin can handle requests from all edge locations.
• Security Groups and Firewalls: Ensure your security groups and firewall settings allow incoming connections from all CloudFront IP ranges.

Examine CloudFront and ELB Logs:

• CloudFront Logs: Enable CloudFront logging to get detailed logs of requests and errors. This can provide insights into what might be causing the 504 errors.
• ELB Logs: Double-check ELB logs for any signs of issues that might not have been apparent initially. Look for any anomalies in the logs that could hint at connectivity or performance issues.

Test Direct Origin Access:

• Bypass CloudFront: Access your origin server directly (bypassing CloudFront) from different locations to see if the issue persists. This can help isolate whether the problem is with CloudFront or the origin.

Review Error Messages:

• originCommError: This error indicates a communication problem between CloudFront and the origin. This could be due to origin server latency, network issues, or misconfiguration in CloudFront.

Settings and Configurations to Review

CloudFront Cache Behavior:

• Ensure your cache behavior settings are correctly configured for the paths in question.
• Verify the Origin Protocol Policy and Cache Based on Selected Request Headers are appropriately set.

Origin Settings:

• Origin Domain Name: Ensure the domain name is correct and resolvable.
• Custom Headers: Check if any custom headers might be affecting the response.
• SSL/TLS Settings: Verify that SSL/TLS settings between CloudFront and the origin are correctly configured.

TTL Configuration:

• Consider setting a small TTL instead of 0 to reduce the load on your origin while still ensuring relatively fresh content.

Example Configurations to Adjust

{
  "DistributionConfig": {
    "Origins": {
      "Items": [
        {
          "Id": "origin-1",
          "DomainName": "example.com",
          "OriginPath": "",
          "CustomHeaders": {
            "Quantity": 0
          },
          "S3OriginConfig": {
            "OriginAccessIdentity": ""
          },
          "CustomOriginConfig": {
            "HTTPPort": 80,
            "HTTPSPort": 443,
            "OriginProtocolPolicy": "https-only",
            "OriginSslProtocols": {
              "Quantity": 1,
              "Items": [
                "TLSv1.2"
              ]
            },
            "OriginReadTimeout": 30,  // Increase this value
            "OriginKeepaliveTimeout": 5
          }
        }
      ],
      "Quantity": 1
    }
  }
}

Hi everyone,

Thanks you guys helping first.

Let me provide more info first: The site is running normally as 4-5 years, but some pages with all TTL 0 become 504 suddenly few days ago. Hence, I created test page "halo.php" with all TTL 0 for testing.

As I'm a programmer, im not familiar with AWS infrastructure/environment too much. I may be ask the stupid questions below:

1. I did an experimental - I shut down the web server apache. All other pages show "502 bad gateway" immediately. However, the page "halo.php" with all TTL 0 still show 504 error (HKG1-P2). Is it possible or normal?
2. For "halo.php" 504 error page, I also can't find any requests in web server access logs. Is it represent this request can't reach the web server? This request was dropped connection by "unknown reason" before reaching the web server?

Thanks Wayne