Skip to content

Pull Aurora Logs storage consumption in CloudWatch

0

Hi, We seem to be massively over logging with Aurora. I think this is because of log_replication_commands. I'd like to get an idea of what we are currently storing across all our accounts. I have written the below using boto3 though the values only seem to match logs that are visible within the instance. When I look at CloudWatch, I can see we have stored 100GB+ of logs, is there a way I can return this value using Boto3?

sum([log['Size'] for log in rds_client.describe_db_log_files(DBInstanceIdentifier=db_instance)['DescribeDBLogFiles']])

1 Answer
0
Accepted Answer

The issue you're experiencing is that the describe_db_log_files() method only returns information about log files that are still accessible on the DB instance itself, not the logs that have been exported and stored in CloudWatch Logs. When Aurora publishes logs to CloudWatch Logs, they become separate from the instance's local log files.

To get the actual storage consumption of your Aurora logs in CloudWatch Logs, you'll need to use the CloudWatch Logs API instead of the RDS API. Specifically, you should use the describe_log_groups() method from the CloudWatch Logs client to get storage information.

Here's the approach:

  1. Use the CloudWatch Logs client (not RDS client)
  2. Look for log groups with the prefix /aws/rds/cluster/ followed by your cluster names
  3. The describe_log_groups() response includes a storedBytes field that shows the total bytes stored for each log group

Here's a code example:

import boto3

cloudwatch_logs = boto3.client('logs')

# Get all RDS-related log groups
paginator = cloudwatch_logs.get_paginator('describe_log_groups')
total_bytes = 0

for page in paginator.paginate(logGroupNamePrefix='/aws/rds/'):
for log_group in page['logGroups']:
total_bytes += log_group.get('storedBytes', 0)
print(f"{log_group['logGroupName']}: {log_group.get('storedBytes', 0) / (1024**3):.2f} GB")

print(f"\nTotal storage: {total_bytes / (1024**3):.2f} GB")

This will give you the actual CloudWatch Logs storage consumption across all your Aurora clusters, which should match the 100GB+ you're seeing in the console.
Sources
Publishing database logs to Amazon CloudWatch Logs - Amazon Aurora
Publishing Aurora PostgreSQL logs to Amazon CloudWatch Logs - Amazon Aurora
Publishing Amazon Aurora MySQL logs to Amazon CloudWatch Logs - Amazon Aurora
Monitoring log events in Amazon CloudWatch - Amazon Aurora

answered 5 months ago

AWS
EXPERT

reviewed 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.