I want to know how to write a filter expression that I can use in the amazon-cloudwatch-agent config to filter out JSON log entries that match a particular key-value. I am trying to add a cloudwatch config to my elasticbeanstalk which will exclude health check JSON logs from being published to Cloudwatch.
Jun 10 07:02:38 ip-x-y-p-q myapp[pid]:
{
"req": {
"id": 7,
"method": "GET",
"url": "/health",
"query": {},
"params": {},
...
},
}
Specifically I want to exclude logs with the key-value "url": "/health"
. My elasticbeanstalk config is like this:
files:
"/opt/aws/amazon-cloudwatch-agent/bin/config.json":
mode: "00600"
owner: root
group: root
content: |
{
"agent": {
"metrics_collection_interval": 10
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/myapp.stdout.log",
"log_group_name": "`{ "Fn::Join": [ "/", [ "/aws/elasticbeanstalk", { "Ref": "AWSEBEnvironmentName" }, "var/log/myapp.log" ] ] }`",
"log_stream_name": "{instance_id}",
"retention_in_days": 7,
"filters": [
{
"type": "exclude",
"expression": "?"
}
]
}
]
}
}
}
}
container_commands:
start_cloudwatch_agent:
command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
But my attempts seem to either exclude all logs so far. I have tried
- ""url":\s"/health""
- ""url": "/health""
- "{ $.req.url = "/health"}"
In some cases it seems the cloudwatch agent itself doesn't start (presumably because of errors in the filter expression), when checked using sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status