EKS Blueprint deployment of AWS for fluent bit parses JSON to string

Hi,

AWS enabled the support for monitoring JSON-formatted logs with CloudWatch Logs which enable you to create graphs and receive notifications when your JSON-formatted log events contain terms or match conditions that you choose. [1] AWS also supports logging from the pods directly to CloudWatch. [2]

However, parsing the files from the container to the CloudWatch is done by Fluent-bit which always displayed in the CloudWatch as stdout. The suggested solution to make JSON logs work, is to use a parser in your config map as shown in the example below:

kind: ConfigMap
apiVersion: v1
metadata:
  name: xxxx
  namespace: xxxx
  labels:
data:
  parsers.conf: |
    [PARSER]
        Name docker
        Format json
        Time_Key time
  output.conf: |
    [OUTPUT]
        Name cloudwatch
        Match   *
        region xxxx
        log_group_name xxxx
        log_stream_prefix from-fluent-bit-
        auto_create_group true
        log_key log

References: [1] https://aws.amazon.com/about-aws/whats-new/2015/01/20/amazon-cloudwatch-logs-json-log-format-support/ [2] https://docs.amazonaws.cn/en_us/eks/latest/userguide/fargate-logging.html

My attempt (Which total faiiled :D)

Current values file (Which is just the default values):

serviceAccount:
  create: false
  name: aws-for-fluent-bit-sa

cloudWatchLogs:
  enabled: true
  region: eu-west-2
  logGroupName: /oc-prod-backend/worker-fluentbit-logs

this creates a configmap like:

apiVersion: v1
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File /fluent-bit/parsers/parsers.conf
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Merge_Log_Key       data
        Keep_Log            On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Buffer_Size         32k
    [OUTPUT]
        Name                  cloudwatch_logs
        Match                 *
        region                eu-west-2
        log_group_name        /oc-prod-backend/worker-fluentbit-logs
        log_group_template    /aws/eks/fluentbit-cloudwatch/workload/$kubernetes['namespace_name']
        log_stream_prefix     fluentbit-
        log_stream_template   $kubernetes['pod_name'].$kubernetes['container_name']
        auto_create_group     true
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: aws-for-fluent-bit
    meta.helm.sh/release-namespace: aws-for-fluent-bit
  creationTimestamp: "2023-05-05T13:39:08Z"
  labels:
    app.kubernetes.io/instance: aws-for-fluent-bit
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: aws-for-fluent-bit
    app.kubernetes.io/version: 2.31.11
    helm.sh/chart: aws-for-fluent-bit-0.1.27

tried updating to below as its seems the closest thing i can get.

serviceAccount:
  create: false
  name: aws-for-fluent-bit-sa

cloudWatchLogs:
  enabled: true
  region: eu-west-2
  logGroupName: /oc-prod-backend/worker-fluentbit-logs

service:
  extraParsers: |
    [PARSER]
        Name     docker
        Format   json
        Time_Key time

which updates the configmap to:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File /fluent-bit/parsers/parsers.conf
        Parsers_File /fluent-bit/etc/parser_extra.conf
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Merge_Log_Key       data
        Keep_Log            On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Buffer_Size         32k
    [OUTPUT]
        Name                  cloudwatch_logs
        Match                 *
        region                eu-west-2
        log_group_name        /oc-prod-backend/worker-fluentbit-logs
        log_group_template    /aws/eks/fluentbit-cloudwatch/workload/$kubernetes['namespace_name']
        log_stream_prefix     fluentbit-
        log_stream_template   $kubernetes['pod_name'].$kubernetes['container_name']
        auto_create_group     true
  parser_extra.conf: |-
    [PARSER]
        Name     docker
        Format   json
        Time_Key time
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: aws-for-fluent-bit
    meta.helm.sh/release-namespace: aws-for-fluent-bit
  creationTimestamp: "2023-05-05T13:39:08Z"

which stops the logs coming in and gets an log message like :

[2023/07/25 13:27:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream resque-workers-opensearch-9cc86684d-c86dh.resque-workers-opensearch in log group /aws/eks/fluentbit-cloudwatch/workload/resque-workers-opensearch
[2023/07/25 13:27:56] [error] [http_client] broken connection to logs.eu-west-2.amazonaws.com:443 ?
[2023/07/25 13:27:56] [error] [http_client] broken connection to logs.eu-west-2.amazonaws.com:443 ?
[2023/07/25 13:27:56] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to create log stream
[2023/07/25 13:27:56] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2023/07/25 13:27:56] [ warn] [engine] failed to flush chunk '1-1690291675.655229071.flb', retry in 7 seconds: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0)