- Newest
- Most votes
- Most comments
As the bot wrote, the dashboard views in the console are only available via the console. You can calculate the equivalent data programmatically from CloudFront standard logs, which you mentioned having enabled. I believe that's also what CloudFront does internally to produce the console views, even if it might precalculate and consolidate data on the fly for quicker analysis. Athena is able to process huge amounts of data, although I don't know how immense your volumes are.
I appreciate this might not be useful to you, since you mentioned preferring to avoid analysing logs, but since this can all be done rather easily with standard AWS services, it might still be useful to others later.
First, Athena needs to be set up to analyse the logs. Before using Athena for the first time, you should create a dedicated S3 bucket for Athena's query results in the same region where you created the S3 bucket for the CloudFront logs. You should also run Athena in the same region for the best performance and lowest costs.
When the result bucket is created, open the Athena console, and in the Query editor, enter the result bucket's name and optional prefix (subfolder) on the Settings tab in "Query result location".
Next, in the Query editor, create a new database for the logs (for example: create database cloudfront_logs
) or choose an existing database. Create a table for the CloudFront logs as per the instructions in this documentation article: https://docs.aws.amazon.com/athena/latest/ug/create-cloudfront-table-standard-logs.html. You'll need to replace the S3 location in the documentation example with the name of your log bucket and the prefix under which the logs for the distribution you want are sent.
If you'd like, you can also use Athena's partition projection feature to use a single table conveniently for multiple distributions, but that isn't strictly necessary, because you can also just create a separate table for each distribution.
Once the table is created, it's useful to test that it works with a simple SQL query in Athena, such as:
select * from cloudfront_standard_logs limit 100
Finally, you can have Athena calculate similar data as that shown by the Popular pages console view with the SQL query below. It's set to read logs for the past 30 days, so adjust to your liking:
select cs_uri_stem as "Object", count(*) as Requests, count(case when x_edge_response_result_type='Hit' then 1 end) as Hits, cast( round( cast(count(case when x_edge_response_result_type in ('Hit','RefreshHit') then 1 end) as decimal(18, 6)) / cast(count(*) as decimal(18, 6)) * cast(100 as decimal(18, 6)), 2 ) as decimal(18, 2) ) as HitsPct, count(case when x_edge_response_result_type='Miss' then 1 end) as Misses, sum(case when x_edge_response_result_type='Miss' then sc_bytes end) as BytesFromMisses, sum(sc_bytes) as TotalBytes, count( case when (sc_status=200 and x_edge_result_type='Error') or (x_edge_response_result_type!='Error' and x_edge_result_type='Error') then 1 end ) as IncompleteDownloads, count(case when sc_status between 200 and 299 then 1 end) as "2xx", count(case when sc_status between 300 and 399 then 1 end) as "3xx", count(case when sc_status between 400 and 499 then 1 end) as "4xx", count(case when sc_status between 500 and 599 then 1 end) as "5xx" from cloudfront_standard_logs where x_edge_result_type!='Redirect' and date>=date_add('day', -30, now()) group by cs_uri_stem order by Requests desc, "Object" asc
Athena stores its query results in the result bucket you specified, which you can ingest with your custom program and optionally cache them to reduce the need to scan the logs repeatedly. You can also call Athena programmatically to execute the query.
Unfortunately, there isn't a straightforward official API or command-line method to programmatically retrieve the Popular Objects report from CloudFront. The report you're referring to in the CloudFront console under Reports & Analytics is generated internally by CloudFront and is not directly accessible through AWS APIs or CLI tools.
However, there are a few alternative approaches you could consider:
-
CloudFront Usage Charts: You can use CloudFront Usage Charts to track trends in data transfer and requests for your distributions. These charts provide information on the number of HTTP/HTTPS requests, data transferred over HTTP/HTTPS, and data transferred from CloudFront to your origin. While not as detailed as the Popular Objects report, it can give you some insights into your usage.
-
CloudWatch Metrics: CloudFront sends metrics to Amazon CloudWatch, which you can query programmatically. This won't give you the exact same information as the Popular Objects report, but it can provide useful data on your distribution's performance and usage.
-
Cache Statistics Reports: These reports show detailed cache usage information, which could be helpful in understanding which objects are frequently accessed.
-
AWS Usage Report: This report includes information about CloudFront usage, such as data transfer and request counts. While not as granular as the Popular Objects report, it can provide overall usage statistics.
If you absolutely need the specific data from the Popular Objects report, you might have to resort to less conventional methods like web scraping, as you mentioned. However, these methods are not officially supported and may break if the console interface changes.
Regarding access logs, while they would provide the most detailed information, your concern about their size is valid. If you decide to enable them in the future, you could consider using real-time logs with Amazon Kinesis Data Streams for more manageable and immediate access to log data.
In summary, while there's no direct API for the Popular Objects report, you can use a combination of CloudFront's built-in reporting features and CloudWatch metrics to gather similar usage data for your S3 resources served through CloudFront.
Sources
Reports & Analytics for CDN Performance | Amazon CloudFront
Reports, metrics, and logs - Amazon CloudFront
AWS billing and usage reports for CloudFront - Amazon CloudFront
Relevant content
- asked 2 years ago
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 4 months ago