Failed TimeStream insert using aws sdk on a lambda JVM - Failed when retrieving a required endpoint from AWS.

0

We have a Kotlin (JVM) lambda that we use to insert records into a TimeStream DB. Both are in the same region.

We are using version 2.17.100 or the aws sdk.

When busy, it often fails with a EndpointDiscoveryFailedException (based on the file name in the stack trace) exception with the message "Failed when retrieving a required endpoint from AWS." and the message in the exception cause is "unable to create new native thread". The exception is thrown by software.amazon.awssdk.core.endpointdiscovery.EndpointDiscoveryFailedException$BuilderImpl

I've searched and found some references to the exception being due to Lambda throttling, although that seems weird as there are instances of it occurring when the request rate is less than one request per second (much lower than I'd expect for lambda throttling).

Also, it consistently occurs in Endpoint Discovery while attempting to call TimeStream which to me suggests a TimeStream issue instead of Lambda throttling.

I also found reference to a timestream quota for CRUD APIS being one per second per region. That does seem to be a likely cause, the the exception message is weird AND I've seen the system handle more than one request per second.

Has anybody seen this and do they have an explanation?

Thanks

Jeff
已提问 2 年前1202 查看次数
1 回答
0
已接受的回答

I found the cause - It was a resource/thread starvation issue.

The lambda's runtime was being re-used for multiple requests.

The lambda created a new TimeStreamWriteClient for each request.

TimeStreamWriteClient implements Closable (or AutoClosable) and it needs to be closed as having multiple open instances around uses resources.

I wasn't closing the client. Closing the client after each request - in this case a Kotlin use statement (roughly equivalent to a Java try-with-resources) - appears to have solved this issue.

Jeff
已回答 2 年前
  • I'm not sure if accepting my own answer violates the conventions, but AWS re:Post sent me an email telling me to accept the answer if I though it was right, so I did.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则