Failed TimeStream insert using aws sdk on a lambda JVM - Failed when retrieving a required endpoint from AWS.

0

We have a Kotlin (JVM) lambda that we use to insert records into a TimeStream DB. Both are in the same region.

We are using version 2.17.100 or the aws sdk.

When busy, it often fails with a EndpointDiscoveryFailedException (based on the file name in the stack trace) exception with the message "Failed when retrieving a required endpoint from AWS." and the message in the exception cause is "unable to create new native thread". The exception is thrown by software.amazon.awssdk.core.endpointdiscovery.EndpointDiscoveryFailedException$BuilderImpl

I've searched and found some references to the exception being due to Lambda throttling, although that seems weird as there are instances of it occurring when the request rate is less than one request per second (much lower than I'd expect for lambda throttling).

Also, it consistently occurs in Endpoint Discovery while attempting to call TimeStream which to me suggests a TimeStream issue instead of Lambda throttling.

I also found reference to a timestream quota for CRUD APIS being one per second per region. That does seem to be a likely cause, the the exception message is weird AND I've seen the system handle more than one request per second.

Has anybody seen this and do they have an explanation?

Thanks

Jeff
asked 2 years ago1160 views
1 Answer
0
Accepted Answer

I found the cause - It was a resource/thread starvation issue.

The lambda's runtime was being re-used for multiple requests.

The lambda created a new TimeStreamWriteClient for each request.

TimeStreamWriteClient implements Closable (or AutoClosable) and it needs to be closed as having multiple open instances around uses resources.

I wasn't closing the client. Closing the client after each request - in this case a Kotlin use statement (roughly equivalent to a Java try-with-resources) - appears to have solved this issue.

Jeff
answered 2 years ago
  • I'm not sure if accepting my own answer violates the conventions, but AWS re:Post sent me an email telling me to accept the answer if I though it was right, so I did.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions