Fluent Bit Logs, Kinesis vs OpenSearch (ElasticSearch) Directly

Question

I have Fargate Service with FireLens and Fluent Bit.
I want to send logs to OpenSearch (or ElasticSearch). Two good options:

1. Firelens / FluentBit -> [kinesis plugin](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) with **compression and aggregation** -> Kinesis Data Stream -> Kinesis Firehose with Lambda that decompresses and parses logs -> OpenSearch (or ElasticSearch).
2. Firelens / FluentBit -> [es plugin](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch) -> Open Search (or ElasticSearch).

First version saves bandwidth over the network and custom log parsing can be done.  
But if no custom log parsing is needed, how can I measure that the second solution is better?  
Is it better by the way?

invaderZYM · Answer

While either option could work. Its typically better to air on the side caution and store the logs. You can always have the logs for later use with other services, audits, observability (across your entire stack). To that end, you can use the Kinesis solution for log preservation/parsing, but it also adds more moving parts to your solution. What i would do is

Firelens/Fluent Bit -> CloudWatch -> OpenSearch (or ES)

This gives you the best of both benefits of the solutions you highlighted. Unless there is another reason for Kenisis, i.e. real time analytics for example, this simplifies the pipeline considerably.

Here are some resources to get you started

Firelens/Fluent Bit -> CloudWatch https://aws.amazon.com/blogs/containers/fluent-bit-integration-in-cloudwatch-container-insights-for-eks/

CloudWatch -> OpenSearch https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html.

Fluent Bit Logs, Kinesis vs OpenSearch (ElasticSearch) Directly

Relevant content