How do I stream container logs to CloudWatch in Amazon EKS?

5 minute read
1

I want to learn how to stream logs from containers running in Amazon Elastic Kubernetes Service (Amazon EKS) to CloudWatch Logs using Fluent Bit or Fluentd.

Short description

You can use the external resources Fluent Bit or Fluentd to send logs from your containers to your Amazon CloudWatch logs. For more information on Fluent Bit, see Fluent Bit on the Fluent Bit website. For more information on Fluentd, see Fluentd on the Fluentd website.

Because Fluent Bit is the default log solution for Amazon CloudWatch Container Insights, use Fluent Bit instead of Fluentd. Amazon provides a Fluent Bit container image on Amazon Elastic Container Registry (Amazon ECR). For more information, see AWS for Fluent Bit image repositories for Amazon ECS.

When you set up Fluent Bit as a DaemonSet to send logs to CloudWatch, Fluent Bit creates the following log groups and sources:

  • /aws/containerinsights/Cluster_Name/application: Log source is all log files in /var/log/containers.
  • /aws/containerinsights/Cluster_Name/host: Log source is logs from /var/log/dmesg, /var/log/secure, and /var/log/messages.
  • /aws/containerinsights/Cluster_Name/dataplane: Log source is the logs in /var/log/journal for kubelet.service, kubeproxy.service, and docker.service.

Resolution

Stream container logs that run in your Amazon EKS on an EC2 cluster

To stream containers logs to CloudWatch logs, complete the following steps:

  1. To create a namespace called amazon-cloudwatch, run the following command:

    kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
  2. To create a ConfigMap called fluent-bit-cluster-info, including the cluster name and the Region that you want to send logs to, run the following command.

    ClusterName=<my-cluster-name>
    RegionName=<my-cluster-region>
    FluentBitHttpPort='2020'
    FluentBitReadFromHead='Off'
    [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
    [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
    kubectl create configmap fluent-bit-cluster-info \
    --from-literal=cluster.name=${ClusterName} \
    --from-literal=http.server=${FluentBitHttpServer} \
    --from-literal=http.port=${FluentBitHttpPort} \
    --from-literal=read.head=${FluentBitReadFromHead} \
    --from-literal=read.tail=${FluentBitReadFromTail} \
    --from-literal=logs.region=${RegionName} -n amazon-cloudwatch

    Note: Replace my-cluster-name and my-cluster-region with your cluster's name and Region.

  3. To use Fluent Bit to send logs to an EC2 cluster, run the following Fluent Bit optimized configuration DaemonSet command:

    kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
  4. (Optional) To use Fluent Bit on Amazon Elastic Container Registry (Amazon ECR), run the following aws-for-fluent-bit DaemonSet command:

    kubectl patch ds fluent-bit -n amazon-cloudwatch -p \'{"spec":{"template":{"spec":{"containers":[{"name":"fluent-bit","image":"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest"}]}}}}'
  5. To create AWS Identity and Access Management (IAM) roles for service accounts, run the following eksctl command:

    eksctl create iamserviceaccount \
        --name fluent-bit \
        --namespace amazon-cloudwatch \
        --cluster $CLUSTER \
        --attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
        --approve \
        --override-existing-serviceaccounts

    Note: Replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts.

    For more information, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.

Troubleshoot Fluent Bit deployment

To troubleshoot the Fluent Bit deployment, complete the following steps:

  1. To list the pod names in amazon-cloudwatch namespace, run the following command:

    kubectl get pods -n amazon-cloudwatch

    Note: The pod name returns as fluent-bit-*****.

  2. To check the events output, run the following command:

    kubectl describe pod POD_NAME -n amazon-cloudwatch

    Note: Replace POD_NAME with the name of the pod.

  3. To check the logs, run the following command:

    kubectl logs pod-name -n amazon-cloudwatch

    Note: Replace POD_NAME with the name of the pod.

Delete Fluent Bit deployment

To delete Fluent Bit deployment, run the following command:

kubectl delete configmap fluent-bit-cluster-info -n amazon-cloudwatch 
kubectl delete -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

Stream container logs that run in your Amazon EKS on an AWS Fargate cluster

Amazon EKS on an AWS Fargate cluster has a built-in log router based on Fluent Bit. Because the cluster has a built-in log router, Amazon automatically runs Fluent Bit for you. For more details, see Fargate logging.

To stream containers logs to CloudWatch logs, complete the following steps:

  1. To create a dedicated Kubernetes namespace called aws-observability, run the following command:

    cat <<EOF > aws-observability-namespace.yaml
    kind: Namespace
    apiVersion: v1
    metadata:
      name: aws-observability
      labels:
        aws-observability: enabled
    EOF
    
    kubectl apply -f aws-observability-namespace.yaml
  2. To create a ConfigMap with a Fluent Conf data value to stream container logs to CloudWatch logs, run the following command:

    cat <<EOF > aws-logging-cloudwatch-configmap.yaml
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: aws-logging
      namespace: aws-observability
    data:
      output.conf: |
        [OUTPUT]
            Name cloudwatch_logs
            Match   *
            region region-code
            log_group_name fluent-bit-cloudwatch
            log_stream_prefix from-fluent-bit-
            auto_create_group true
            log_key log
    
      parsers.conf: |
        [PARSER]
            Name crio
            Format Regex
            Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L%z
      
      filters.conf: |
         [FILTER]
            Name parser
            Match *
            Key_name log
            Parser crio
    EOF
    
    kubectl apply -f aws-logging-cloudwatch-configmap.yaml
  3. To create an IAM policy, use the CloudWatch IAM policy. Then, attach the IAM policy to the pod execution role that is specified for your AWS Fargate profile.

    To download the IAM policy to your computer, run the following example command:

    curl -o permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

    To create an IAM policy from the policy file that you downloaded, run the following command:

    aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

    To attach the IAM policy to the pod execution role that is specified for your AWS Fargate profile, replace 111122223333 with your account ID. See the following example command:

    aws iam attach-role-policy \
    --policy-arn arn:aws:iam::111122223333:policy/eks-fargate-logging-policy \
    --role-name your-pod-execution-role

    For more information, see Troubleshooting.

Turn off streaming logs for EKS on Fargate pods

To turn off streaming logs for your EKS on Fargate pods, run the following command:

kubectl delete namespace aws-observability

Delete pods and redeploy them after you delete the aws-observability namespace.

Related information

Container Insights

4 Comments

Thanks for this - was able to get the logs created for our EC2 nodes.

However, we have only 2 nodes, but it is trying to schedule the creation of 11 pods - any idea why this might be the case, and how I should patch this to a suitable number?

Regards,

Rory

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

Is there a helm chart I can use to deploy this, rather than maintaining the YAML manifests myself?

I would also like to deploy separate daemonsets in different namespaces to allow me to log application logs to individual AWS accounts. Can you expand the explanation to cover this case - e.g. is the amazon-cloudwatch namespace still necessary?

replied a month ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a month ago