application log management with AWS cloud watch agent


I am puzzled by how to effectively transfer application logs to CloudWatch. I have 12 microservices, each of which generates five different types of logs located at /var/log/application/service-name. My application is deployed on an AWS ECS cluster, consisting of two large instances capable of hosting all 12 microservices, with each instance accommodating six services.

The primary challenge lies in managing the logs. I aim to organize them such that each type of log resides within a separate log group, and each of the 12 services has its logs stored as individual log streams. In essence, this necessitates creating log groups and 12 log streams per log group.

I have attempted to address this issue by installing the CloudWatch agent and creating a log configuration file. However, I encountered difficulties in properly segregating the logs.

Ideally, I require log group names structured as env_name-log_type.log and log stream names formatted as service_name-instance_id.log.

Please provide suggestions on how to achieve this segmentation.

1 Answer

To achieve the desired log segmentation structure, you can configure the CloudWatch agent to send logs to CloudWatch Logs using log groups and log streams.

Some suggestions:

Configure the CloudWatch agent on each EC2 instance using a configuration file that specifies the log file locations and log group/stream formats.

For the log group name, use the format env_name-log_type.log (e.g. prod-access.log )

For each microservice, define a log stream with the name format service_name-instance_id.log (e.g. paymentservice-i-0123456.log )

The agent will collect logs from the specified files and write them to CloudWatch Logs using the defined log groups and streams.

You can define multiple outputs in the configuration to send the same logs to other destinations like S3 if required.

Logs from all instances will be organized by service and type in the CloudWatch console for easy analysis and troubleshooting.

Let me know if any part needs more explanation or if you have additional questions!

profile picture
answered 2 months ago
  • Hi @Giovanni Lauria thanks for answering... I will create a CloudWatch configuration file within the EC2 user data script. However, I cannot guarantee that the specific service will be present on the same instance consistently, especially after restarts. There is a possibility that this service may be switched to another EC2 instance. Please advise on how to dynamically configure the log configuration file so that it sends the appropriate logs to the correct log groups whenever there is a service swap.

    Furthermore, I am curious about the behavior of the CloudWatch agent if there are no logs present in the specified folder.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions