AWS Cloudwatch Logs to Metrics parsing issue

From your sample event, it seems, you are trying to parse json message, however this is not json as it has strings before and after, if you go to any json formatter and format it, you'll see the problems with this.

I'd suggest you two things here:

1. Get the logs created in a way that you make sure that it has valid json format.
2. When you create cloudwatch metric filter, for json type logs, left side of filter would be key and right side of filter would be value.

Based on what you mentioned, you are attempting to filter those logs which has total_count=57. Metric filter works in a way, that from all the log streams of a log group, it would match key-value defined in a filter and then would return those messages which will match that key-value pair.

If I misunderstood your requirement and you are trying to get total_count from all log events, that's not how filtering works. In filter, you have to match the key value pair not the key finder and then expect it to return value for those key.

So to demonstrate it, I converted your log to json and then put it a single line(test event work with single line only), here's the snapshot of what I did and it's outcome.

Sample log event messages for your reference:

   {"service": "test", "event": "batch_indexing_job","index": "data-object","total_count": 58,"error_count": 0,"duration": 0.14786760200513527}
   {"service": "test", "event": "batch_indexing_job","index": "data-object","total_count": 57,"error_count": 0,"duration": 0.14786760200513527}

For more examples, refer Cloudwatch metric and filter pattern syntax

Hope you find this useful.

Abhishek

Hi,

yes i know the log isnt in json but the i thought the actual log was (if that makes sense!) The logs are coming from containerinsight so each log line looks like:

{
    "log": "2023-07-06T18:53:55.012247992Z stdout F {\"service\":\"IndexWorker\",\"event\":\"batch_indexing_job\",\"index\":\"data-object\",\"total_count\":1,\"error_count\":0,\"duration\":0.03670474399405066}",
    "kubernetes": {
        "pod_name": "test-9cc86684d-wwmwt",
        "namespace_name": "batch-index",
        "pod_id": "251b573f-5165-4c2a-9ea6-3bbe1168664a",
        "labels": {
            "app": "batch-index",
        },
        "host": "ip-10-4-94-159.eu-west-2.compute.internal",
        "container_name": "batch-index",
        "docker_id": "792d575a3591d60560872075ae7beef2512f584679120ae0b7decc1305398a83",
        "container_hash": "xxxxx.dkr.ecr.eu-west-2.amazonaws.com/test@sha256:6b91d9983af9318b96a40f197b8d9d3631479cb6a8903f329250adc06d2a0767",
        "container_image": "xxxxx.dkr.ecr.eu-west-2.amazonaws.com/openc-master:2023-06-30"
    }
}

Hense me thinking i can still parse log via the JSON $log. I just wasnt sure if it was possible to return the total_count value based on the messy string value inside the log. I appreciate yes it would be easy if the log line was json as well, im trying to see if it can be done without that change?

So! ive got the logs into full json now..

To clarify what im trying to do is not record if a specific value is in the key of error_count. But actually record the value for each log error_count..

I thought you could do this natively in cloudwatch is that not the case and i need to create a lambda to parse the logs and create a metric?