How can I filter Enhanced Monitoring CloudWatch logs to generate automated custom metrics for Amazon RDS?

4 minute read

I want to filter Enhanced Monitoring Amazon CloudWatch Logs to generate automated custom metrics for Amazon Relational Database Service (Amazon RDS).

Short description

Enhanced Monitoring for Amazon RDS provides the following OS metrics:

  • Free Memory
  • Active Memory
  • Swap Free
  • Processes Running
  • File System Used

Use these metrics To understand the environment's performance, you can use these OS metrics. Amazon CloudWatch Logs ingest these metrics as log entries. You can use CloudWatch to create alarms based on metrics. These alarms run actions, and you can publish these metrics from within your infrastructure, device, or application into CloudWatch as a custom metric. To use Enhanced Monitoring and CloudWatch to automate tasks, create a custom metric for the CloudWatch Logs Amazon RDS ingested date from the Enhanced Monitoring metrics.


Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent version of the AWS CLI.

Use filters on a log group to create a custom metric

Note: These steps require that you turn on Enhanced Monitoring in your Amazon RDS DB instance. For more information, see Monitoring metrics in an Amazon RDS instance.

  1. Turn on Enhanced Monitoring.

  2. Open the Amazon CloudWatch console, and then choose Log groups from the navigation pane.

  3. Filter for RDSOSMetrics from the list of Log Groups, and then choose Actions.

  4. Choose Create Metric Filter, and then enter a Filter Pattern term for your Amazon RDS DB instance. For example:

    DISK WRITE KBPS GREATER THAN OR EQUAL TO 10 : { $.diskIO[0].writeKbPS >= 10.00 } 
    DISK READ KBPS GREATER THAN OR EQUAL TO 10 : { $.diskIO[0].readKbPS >= 10.00 }
    DISK WRITE IOPS PS GREATER THAN 15 : { $.diskIO[0].writeIOsPS > 15.00 }
    DISK READ IOPS PS GREATER THAN 15 : { $.diskIO[0].readIOsPS > 15.00 }
    FREE MEMORY LESS THAN OR EQUAL TO 200000 KB : { $ <= 200000 }

    Note: Update the values to the values that suit your environment.

  5. Choose Log Data, and then choose Test Pattern.

  6. Choose Next.

  7. Enter a Filter Name, and then choose a Metric Namespace and Metric Name.

  8. Enter a Metric Value. For example, $.cpuUtilization.idle. Then, choose Create Filter.

    For each additional metric that you add, repeat these steps.

Automate your metrics

To automate the process of monitoring the more than 60 monitoring metrics of each Amazon RDS DB instance, use a script. To see an example of a script that uses an Amazon RDS for MySQL DB instance, see rds-support-tools on the GitHub website. The example script works with the following engines:

  • MySQL
  • MariaDB
  • Amazon Aurora MySQL-Compatible Edition
  • Amazon Aurora PostgreSQL-Compatible Edition
  • PostgreSQL

Specify the Amazon RDS DB instance with Enhanced Monitoring turned on and the namespace where you want these metrics to reside. Optionally, add the names of the metrics and the AWS Region. If you don't specify any optional fields, then the script considers all the metrics to publish. Then, the script checks the .aws/config file for the default AWS Region that's set for the AWS CLI.

Note: The script doesn't create metrics for the process list. Depending on the process you want to display, you might need to create filters manually.

Use the following pattern to specify the names of the metrics:


The following sample code illustrates a call to turn on this metric for cpuUtilization.idle and diskIO.readKbPS:

python --rds\_instance mysqltest --namespace MySQL --metrics\_to\_filter cpuUtilization.idle diskIO.readKb

You can then create alarms for these custom metrics. For more information, see Using Amazon CloudWatch alarms.

Related information

Monitoring Amazon RDS log files

Using metric filters to match terms and extract values from JSON log events

AWS OFFICIALUpdated 8 months ago