AWS Lambda failing getaddrinfo EBUSY for too many open files??? (none opened AFAIK)

0

Recently, our AWS Lambda microservices have been failing intermittently due to OS error: EBUSY, getaddrinfo. This caused a function timeout (in DNS call by Redis client, namely).

Because of these intermittent failures, I created a wrapper function whose sole responsibility is to call another function whose timeout is shorter than this.

This does not work either, now the LambdaClient invoke fails for the same reason!

import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
export const handler = async (event) => {
  const lambda = new LambdaClient();
  try {
    var params = {
        FunctionName: '<anotherfunction>', // the lambda function we are going to invoke
        Payload: JSON.stringify(event)
    };
    const command = new InvokeCommand(params);
    let ret = await lambda.send(command);
    
    if (ret && ctx.Payload) {
        let payload = JSON.parse(Buffer.from(ret.Payload));
        if (payload.statusCode == 200) {
          return payload;
        }
    }
  } catch (e) {
    console.error(e);
  }
...  

The failure I get is:

ERROR	Error: getaddrinfo EBUSY lambda.eu-west-1.amazonaws.com
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
  errno: -16,
  code: 'EBUSY',
  syscall: 'getaddrinfo',
  hostname: 'lambda.eu-west-1.amazonaws.com',
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

As far as I know, getaddrinfo EBUSY error comes from underlying OS, and is caused by too many open files. Evidence for that is that I got rid of it (for a while) by changing the function memory parameters a bit so the funcrtion was moved to another tenant.

Is there anything I can do to prevent this? Is there anything AWS can do to prevent this?

BR, Joni

2回答
0

Hi,

A common cause of EBUSY getaddrinfo is running out of file descriptors. This may come from underlying the kernel shared by all the Lambda containers.

So, yes, not much that you can do!

Unless your Lambda opens lots of files or tcp sockets in parallel.... which is probably not the case. You may reach such a case with tcp sockets if your Lambda has high throughput with lots of parallel executions of it on same server: reducing tcp sessions to a very minimum may solve your issue because it will free descriptors.

Best,

Didier

profile pictureAWS
エキスパート
回答済み 1ヶ月前
  • So what are our options? Ditch AWS Lambda? This causes severe downgrade in our production environment.

0

I don't have a specific answer, but I hope this will be helpful.

https://repost.aws/knowledge-center/lambda-intermittent-consistent-dns-error

profile picture
エキスパート
shibata
回答済み 1ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ