Failed TimeStream insert using aws sdk on a lambda JVM - Failed when retrieving a required endpoint from AWS.

0

We have a Kotlin (JVM) lambda that we use to insert records into a TimeStream DB. Both are in the same region.

We are using version 2.17.100 or the aws sdk.

When busy, it often fails with a EndpointDiscoveryFailedException (based on the file name in the stack trace) exception with the message "Failed when retrieving a required endpoint from AWS." and the message in the exception cause is "unable to create new native thread". The exception is thrown by software.amazon.awssdk.core.endpointdiscovery.EndpointDiscoveryFailedException$BuilderImpl

I've searched and found some references to the exception being due to Lambda throttling, although that seems weird as there are instances of it occurring when the request rate is less than one request per second (much lower than I'd expect for lambda throttling).

Also, it consistently occurs in Endpoint Discovery while attempting to call TimeStream which to me suggests a TimeStream issue instead of Lambda throttling.

I also found reference to a timestream quota for CRUD APIS being one per second per region. That does seem to be a likely cause, the the exception message is weird AND I've seen the system handle more than one request per second.

Has anybody seen this and do they have an explanation?

Thanks

Jeff
질문됨 2년 전1202회 조회
1개 답변
0
수락된 답변

I found the cause - It was a resource/thread starvation issue.

The lambda's runtime was being re-used for multiple requests.

The lambda created a new TimeStreamWriteClient for each request.

TimeStreamWriteClient implements Closable (or AutoClosable) and it needs to be closed as having multiple open instances around uses resources.

I wasn't closing the client. Closing the client after each request - in this case a Kotlin use statement (roughly equivalent to a Java try-with-resources) - appears to have solved this issue.

Jeff
답변함 2년 전
  • I'm not sure if accepting my own answer violates the conventions, but AWS re:Post sent me an email telling me to accept the answer if I though it was right, so I did.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠