Enabling CORS policy for public S3 object delivered through CloudFront

I have an S3 object that should be publicly accessible but only from specified domains. The object is accessed through a CloudFront custom url.

I have tried the below steps:

1. Set S3 bucket permission to Public
2. Set 'Origin request policy name' = 'Managed-CORS-S3Origin behavior' in CloudFront distribution according to this post on aws knowledge center so CloudFront passes origin
3. Added CORS policy to S3 bucket as below. https://www.example.com replaced with domain I want to provide access to.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "https://www.example.com"
        ],
        "ExposeHeaders": [
            "Access-Control-Allow-Origin"
        ]
    }
]

While I am able to access the object from specified domain, I am also able to access it from domains NOT specified under AllowedOrigins in the CORS policy.

Looking for some direction on how best to fix this so access is restricted to specified domains only.

Thanks!