I want to learn how to stream logs from containers running in Amazon Elastic Kubernetes Service (Amazon EKS) to CloudWatch Logs using Fluent Bit or Fluentd.
Short description
You can use the external resources Fluent Bit or Fluentd to send logs from your containers to your Amazon CloudWatch logs. For more information on Fluent Bit, see Fluent Bit on the Fluent Bit website. For more information on Fluentd, see Fluentd on the Fluentd website.
Because Fluent Bit is the default log solution for Amazon CloudWatch Container Insights, use Fluent Bit instead of Fluentd. Amazon provides a Fluent Bit container image on Amazon Elastic Container Registry (Amazon ECR). For more information, see AWS for Fluent Bit image repositories for Amazon ECS.
When you set up Fluent Bit as a DaemonSet to send logs to CloudWatch, Fluent Bit creates the following log groups and sources:
- /aws/containerinsights/Cluster_Name/application: Log source is all log files in /var/log/containers.
- /aws/containerinsights/Cluster_Name/host: Log source is logs from /var/log/dmesg, /var/log/secure, and /var/log/messages.
- /aws/containerinsights/Cluster_Name/dataplane: Log source is the logs in /var/log/journal for kubelet.service, kubeproxy.service, and docker.service.
Resolution
Stream container logs that run in your Amazon EKS on an EC2 cluster
To stream containers logs to CloudWatch logs, complete the following steps:
-
To create a namespace called amazon-cloudwatch, run the following command:
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
-
To create a ConfigMap called fluent-bit-cluster-info, including the cluster name and the Region that you want to send logs to, run the following command.
ClusterName=<my-cluster-name>
RegionName=<my-cluster-region>
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch
Note: Replace my-cluster-name and my-cluster-region with your cluster's name and Region.
-
To use Fluent Bit to send logs to an EC2 cluster, run the following Fluent Bit optimized configuration DaemonSet command:
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
-
(Optional) To use Fluent Bit on Amazon Elastic Container Registry (Amazon ECR), run the following aws-for-fluent-bit DaemonSet command:
kubectl patch ds fluent-bit -n amazon-cloudwatch -p \'{"spec":{"template":{"spec":{"containers":[{"name":"fluent-bit","image":"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest"}]}}}}'
-
To create AWS Identity and Access Management (IAM) roles for service accounts, run the following eksctl command:
eksctl create iamserviceaccount \
--name fluent-bit \
--namespace amazon-cloudwatch \
--cluster $CLUSTER \
--attach-policy-arn "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" \
--approve \
--override-existing-serviceaccounts
Note: Replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts.
For more information, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.
Troubleshoot Fluent Bit deployment
To troubleshoot the Fluent Bit deployment, complete the following steps:
-
To list the pod names in amazon-cloudwatch namespace, run the following command:
kubectl get pods -n amazon-cloudwatch
Note: The pod name returns as fluent-bit-*****.
-
To check the events output, run the following command:
kubectl describe pod POD_NAME -n amazon-cloudwatch
Note: Replace POD_NAME with the name of the pod.
-
To check the logs, run the following command:
kubectl logs pod-name -n amazon-cloudwatch
Note: Replace POD_NAME with the name of the pod.
Delete Fluent Bit deployment
To delete Fluent Bit deployment, run the following command:
kubectl delete configmap fluent-bit-cluster-info -n amazon-cloudwatch
kubectl delete -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
Stream container logs that run in your Amazon EKS on an AWS Fargate cluster
Amazon EKS on an AWS Fargate cluster has a built-in log router based on Fluent Bit. Because the cluster has a built-in log router, Amazon automatically runs Fluent Bit for you. For more details, see Fargate logging.
To stream containers logs to CloudWatch logs, complete the following steps:
-
To create a dedicated Kubernetes namespace called aws-observability, run the following command:
cat <<EOF > aws-observability-namespace.yaml
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
EOF
kubectl apply -f aws-observability-namespace.yaml
-
To create a ConfigMap with a Fluent Conf data value to stream container logs to CloudWatch logs, run the following command:
cat <<EOF > aws-logging-cloudwatch-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region region-code
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group true
log_key log
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
EOF
kubectl apply -f aws-logging-cloudwatch-configmap.yaml
-
To create an IAM policy, use the CloudWatch IAM policy. Then, attach the IAM policy to the pod execution role that is specified for your AWS Fargate profile.
To download the IAM policy to your computer, run the following example command:
curl -o permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json
To create an IAM policy from the policy file that you downloaded, run the following command:
aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json
To attach the IAM policy to the pod execution role that is specified for your AWS Fargate profile, replace 111122223333 with your account ID. See the following example command:
aws iam attach-role-policy \
--policy-arn arn:aws:iam::111122223333:policy/eks-fargate-logging-policy \
--role-name your-pod-execution-role
For more information, see Troubleshooting.
Turn off streaming logs for EKS on Fargate pods
To turn off streaming logs for your EKS on Fargate pods, run the following command:
kubectl delete namespace aws-observability
Delete pods and redeploy them after you delete the aws-observability namespace.
Related information
Container Insights