Origin Group getting 403 on 404 failover

Question

Hi,  
  
I have set up two different distributions as so:  
  
Distribution A: s3 origin  
Distribution B: custom origin  
  
When I attempt to GET a known resource from distro A, everything works and a 200 OK is returned. When I attempt to GET a known resource from distro B, everything works and a 200 OK is returned. So, **everything is fine** with an s3 origin and a custom origin.  
  
Now, the resource on each does not exist on the other distros. In other words, the resource on distro A is not in distro B and the resource in distro B is not in distro A.  
  
I have then set up a third distribution that has both of these origins configured and an origin group is set up with each origin and the failover criteria of 404 are set. The s3 origin is priority 1 and the custom origin is priority 2. So, looks like this:  
  
Distribution C: origin group  
- priority 1: s3 origin  
- priority 2: custom origin  
- failover criteria: 404 Not Found

When I perform a GET request for a known resource on distro C the priority 1 origin (s3 origin), a 200 OK is returned. When I perform a GET request that I know will 404 (because it is on the custom origin), I get a 403 Forbidden response.  
  
Is anyone able to explain to me what is happening here, am I missing a configuration setting?

fortycreek · Answer

I just want to give an update. I was able to figure out the problem. It seems that the 404 failover was not taking place. In the response from origin group it had a 403 forbidden reply from "AmasonS3". Digging into that I found this nugget of information:  
  
"_If a user doesn’t have s3:ListBucket permissions, then the user gets Access Denied errors for missing objects instead of 404 Not Found errors. Run the head-object AWS CLI command to check if an object exists in the bucket._"  
  
I was able to test that the 403 was from the S3 origin and not failing over by setting an additional origin group failover criteria to include a 403 response. This initially routed to my custom origin.  
  
However, to ensure the correct reason for failing over was due to 404 and not a misconfigured permission, I added the s3:ListBucket permission , removed the 403 criteria, and the failover from s3 origin to custom origin on a 404 works correclty.  
  
**TL;DR** -- Make sure CF has not only s3:GetObject permissions on the S3 origin bucket, but also s3:ListBucket permissions.

Origin Group getting 403 on 404 failover

Relevant content