AWS Lambda failing getaddrinfo EBUSY for too many open files??? (none opened AFAIK)

0

Recently, our AWS Lambda microservices have been failing intermittently due to OS error: EBUSY, getaddrinfo. This caused a function timeout (in DNS call by Redis client, namely).

Because of these intermittent failures, I created a wrapper function whose sole responsibility is to call another function whose timeout is shorter than this.

This does not work either, now the LambdaClient invoke fails for the same reason!

import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
export const handler = async (event) => {
  const lambda = new LambdaClient();
  try {
    var params = {
        FunctionName: '<anotherfunction>', // the lambda function we are going to invoke
        Payload: JSON.stringify(event)
    };
    const command = new InvokeCommand(params);
    let ret = await lambda.send(command);
    
    if (ret && ctx.Payload) {
        let payload = JSON.parse(Buffer.from(ret.Payload));
        if (payload.statusCode == 200) {
          return payload;
        }
    }
  } catch (e) {
    console.error(e);
  }
...  

The failure I get is:

ERROR	Error: getaddrinfo EBUSY lambda.eu-west-1.amazonaws.com
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
  errno: -16,
  code: 'EBUSY',
  syscall: 'getaddrinfo',
  hostname: 'lambda.eu-west-1.amazonaws.com',
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

As far as I know, getaddrinfo EBUSY error comes from underlying OS, and is caused by too many open files. Evidence for that is that I got rid of it (for a while) by changing the function memory parameters a bit so the funcrtion was moved to another tenant.

Is there anything I can do to prevent this? Is there anything AWS can do to prevent this?

BR, Joni

2 Risposte
0

Hi,

A common cause of EBUSY getaddrinfo is running out of file descriptors. This may come from underlying the kernel shared by all the Lambda containers.

So, yes, not much that you can do!

Unless your Lambda opens lots of files or tcp sockets in parallel.... which is probably not the case. You may reach such a case with tcp sockets if your Lambda has high throughput with lots of parallel executions of it on same server: reducing tcp sessions to a very minimum may solve your issue because it will free descriptors.

Best,

Didier

profile pictureAWS
ESPERTO
con risposta un mese fa
  • So what are our options? Ditch AWS Lambda? This causes severe downgrade in our production environment.

0

I don't have a specific answer, but I hope this will be helpful.

https://repost.aws/knowledge-center/lambda-intermittent-consistent-dns-error

profile picture
ESPERTO
shibata
con risposta un mese fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande