I am currently utilizing the log router of EKS Fargate and encountering issues with Multiline Parsing. Despite following the documentation provided for Fluent Bit's multiline parsing at Fluent Bit Docs https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#built-in-multiline-parsers , I have not been successful in resolving the issue.
Could you please guide how to configure this correctly? Below is my current configuration for handling logs:
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "true" # Set to true to ship Fluent Bit process logs to CloudWatch.
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser cri
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name rewrite_tag
Match kube.*
Rule $kubernetes['namespace_name'] ^pe-stg-nginx-ingress$ nginx-ingress false
Rule $kubernetes['namespace_name'] ^pe-stg-fg$ app false
output.conf: |
[OUTPUT]
Name es
Match nginx-ingress
Index ***
Host ***
HTTP_User ***
HTTP_Passwd ***
Port ***
tls On
tls.verify Off
Suppress_Type_Name On
output.conf: |
[OUTPUT]
Name es
Match app
Index ***
Host ***
HTTP_User ***
HTTP_Passwd ***
Port ***
tls On
tls.verify Off
Suppress_Type_Name On
parsers.conf: |
[PARSER]
Name java
Format cri
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
However, the logs are still not being parsed as expected.
Here is an example of the Java application log that I wish to parse in multiline format:
2024-06-19T03:43:17.747-04:00 INFO 1 --- [ main] .s.b.a.l.ConditionEvaluationReportLogger :
Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-06-19T03:43:17.848-04:00 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pendingRolloverScheduler': Unsatisfied dependency expressed through field 'pendingRolloverService': Error creating bean with name 'pendingRolloverService': Unsatisfied dependency expressed through field 'rolloverService': Error creating bean with name 'rolloverService': Invocation of init method failed
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:767) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:747) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145) ~[spring-beans-6.0.13.jar!/:6.0.13]
I would greatly appreciate your assistance in resolving this matter. Thank you very much for your support.