Hello,
I have created an OpenSearch Ingestion Pipeline for getting ALB Logs from S3 bucket, and pushing them to OpenSearch domain. The S3 source is setup with SQS queue.
When stopping/starting the pipeline it works fine and logs appear in OpenSearch domain. However after an hour or so ingestion stops. Looking at pipeline logs in CloudWatch I can see multiple errors from S3 like:
Error processing from S3: Unable to execute HTTP request: Timeout waiting for connection from pool. Retrying with exponential backoff.
If I stop/start the pipeline it will again work for sometime and then throw these errors again. How can I solve this issue?
Thanks