AWS Kinesis Data Firehose Delivery Error Logs and S3 Backup Anomalies

0

Hello, dear community! I have configured AWS Kinesis Data Firehose to stream data to an HTTP endpoint, with an S3 backup configuration enabled for storing all data in an S3 bucket. Despite this setup, I've encountered a peculiar issue: the delivery error logs in AWS Kinesis Data Firehose remain empty, yet the S3 backup bucket (specifically under the prefix "http-endpoint-failed") shows that some records have been populated.

Under normal circumstances, I would expect that if Kinesis Data Firehose directs records to the S3 backup bucket under the "http-endpoint-failed" prefix, it indicates the occurrence of HTTP delivery errors, which should be logged. However, this has not been the case. Interestingly, without any changes to the configuration, delivery error logs recently started to appear, but only for the most recent errors. The only difference is that I forced the HTTP endpoint to respond with an error manually.

I am speculating whether the buffering hints configuration for both the HTTP endpoint and the S3 backup could be contributing to this issue. Currently, both buffering hints are set to 5MB and 300 seconds. I would appreciate insights or suggestions on why this discrepancy in error logging and data backup behavior is occurring and how it might be resolved.

asked 5 months ago194 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions