Failed TimeStream insert using aws sdk on a lambda JVM - Failed when retrieving a required endpoint from AWS.

0

We have a Kotlin (JVM) lambda that we use to insert records into a TimeStream DB. Both are in the same region.

We are using version 2.17.100 or the aws sdk.

When busy, it often fails with a EndpointDiscoveryFailedException (based on the file name in the stack trace) exception with the message "Failed when retrieving a required endpoint from AWS." and the message in the exception cause is "unable to create new native thread". The exception is thrown by software.amazon.awssdk.core.endpointdiscovery.EndpointDiscoveryFailedException$BuilderImpl

I've searched and found some references to the exception being due to Lambda throttling, although that seems weird as there are instances of it occurring when the request rate is less than one request per second (much lower than I'd expect for lambda throttling).

Also, it consistently occurs in Endpoint Discovery while attempting to call TimeStream which to me suggests a TimeStream issue instead of Lambda throttling.

I also found reference to a timestream quota for CRUD APIS being one per second per region. That does seem to be a likely cause, the the exception message is weird AND I've seen the system handle more than one request per second.

Has anybody seen this and do they have an explanation?

Thanks

Jeff
質問済み 2年前1202ビュー
1回答
0
承認された回答

I found the cause - It was a resource/thread starvation issue.

The lambda's runtime was being re-used for multiple requests.

The lambda created a new TimeStreamWriteClient for each request.

TimeStreamWriteClient implements Closable (or AutoClosable) and it needs to be closed as having multiple open instances around uses resources.

I wasn't closing the client. Closing the client after each request - in this case a Kotlin use statement (roughly equivalent to a Java try-with-resources) - appears to have solved this issue.

Jeff
回答済み 2年前
  • I'm not sure if accepting my own answer violates the conventions, but AWS re:Post sent me an email telling me to accept the answer if I though it was right, so I did.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ