AWS Kinesis Data Firehose Delivery Error Logs and S3 Backup Anomalies

0

Hello, dear community! I have configured AWS Kinesis Data Firehose to stream data to an HTTP endpoint, with an S3 backup configuration enabled for storing all data in an S3 bucket. Despite this setup, I've encountered a peculiar issue: the delivery error logs in AWS Kinesis Data Firehose remain empty, yet the S3 backup bucket (specifically under the prefix "http-endpoint-failed") shows that some records have been populated.

Under normal circumstances, I would expect that if Kinesis Data Firehose directs records to the S3 backup bucket under the "http-endpoint-failed" prefix, it indicates the occurrence of HTTP delivery errors, which should be logged. However, this has not been the case. Interestingly, without any changes to the configuration, delivery error logs recently started to appear, but only for the most recent errors. The only difference is that I forced the HTTP endpoint to respond with an error manually.

I am speculating whether the buffering hints configuration for both the HTTP endpoint and the S3 backup could be contributing to this issue. Currently, both buffering hints are set to 5MB and 300 seconds. I would appreciate insights or suggestions on why this discrepancy in error logging and data backup behavior is occurring and how it might be resolved.

demandé il y a 5 mois199 vues
Aucune réponse

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions