AWS Lambda failing getaddrinfo EBUSY for too many open files??? (none opened AFAIK)

Question

Recently, our AWS Lambda microservices have been failing intermittently due to OS error: EBUSY, getaddrinfo. This caused a function timeout (in DNS call by Redis client, namely).

Because of these intermittent failures, I created a wrapper function whose sole responsibility is to call another function whose timeout is shorter than this.

This does not work either, now the LambdaClient invoke fails for the same reason!

```
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
export const handler = async (event) => {
  const lambda = new LambdaClient();
  try {
    var params = {
        FunctionName: '', // the lambda function we are going to invoke
        Payload: JSON.stringify(event)
    };
    const command = new InvokeCommand(params);
    let ret = await lambda.send(command);
    
    if (ret && ctx.Payload) {
        let payload = JSON.parse(Buffer.from(ret.Payload));
        if (payload.statusCode == 200) {
          return payload;
        }
    }
  } catch (e) {
    console.error(e);
  }
...  
```

The failure I get is: 
```
ERROR	Error: getaddrinfo EBUSY lambda.eu-west-1.amazonaws.com
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
  errno: -16,
  code: 'EBUSY',
  syscall: 'getaddrinfo',
  hostname: 'lambda.eu-west-1.amazonaws.com',
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}
```

As far as I know, getaddrinfo EBUSY error comes from underlying OS, and is caused by too many open files. Evidence for that is that I got rid of it (for a while) by changing the function memory parameters a bit so the funcrtion was moved to another tenant.

Is there anything I can do to prevent this? Is there anything AWS can do to prevent this?

BR, Joni

Ridwaan_M-L · Answer

HI

I noticed your Lambda client is instantiated inside the handler 
const lambda = new LambdaClient();

Is it possible to move this outside the handler and see if it improves the situtation?

```
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
const lambda = new LambdaClient(); //outside handler

export const handler = async (event) => {
  
  try {
    var params = {
        FunctionName: '', // the lambda function we are going to invoke
        Payload: JSON.stringify(event)
    };
    const command = new InvokeCommand(params);
    let ret = await lambda.send(command);
    
    if (ret && ctx.Payload) {
        let payload = JSON.parse(Buffer.from(ret.Payload));
        if (payload.statusCode == 200) {
          return payload;
        }
    }
  } catch (e) {
    console.error(e);
  }
...  
```

Didier_Durand · Answer

Hi,

A common cause of EBUSY getaddrinfo is running out of file descriptors. This may come from underlying the kernel shared by all the Lambda containers.

So, yes, not much that you can do!

Unless your Lambda opens lots of files or tcp sockets in parallel.... which is probably not the case. You may reach such  a case with tcp sockets if your Lambda has high throughput with lots of parallel executions of it on same server: reducing tcp sessions to a very minimum may solve your issue because it will free descriptors.

Best,

Didier

shibata · Answer

I don't have a specific answer, but I hope this will be helpful.

https://repost.aws/knowledge-center/lambda-intermittent-consistent-dns-error

AWS Lambda failing getaddrinfo EBUSY for too many open files??? (none opened AFAIK)

Relevant content