How can I quickly debug a Lambda's failure to mount an EFS access point?

0

I had attached an EFS to a Lambda via an access point. The Lambda resided in the same VPC and same subnets as my EFS mount targets. The EFS had a correct and very permissive IAM file system policy. However, when executing my lambda, I received {"message": "Internal server error"} and no log entries were written to the Cloudwatch Log Group which was attached to my Lambda.

After much trial and error, I discovered that my subnet mount targets had a Security Group that excluded the Lambda function. After fixing the Security Group, the Lambda was able to execute without error, presumably because it was now able to establish a network connection to the EFS.

This resolution took a lot of time, but could have been accelerated had there been points of feedback that explained why the error had occurred. For example, if the Lambda had written something to the CW log group, or if the "file systems" tab of the Lambda had a "test connectivity" button.

Is there somewhere where I could have found the source of the problem faster, but I just missed it? If not, could a better connectivity feedback solution be implemented? Thank you

3 Answers
1
Accepted Answer

It is unclear why there is a problem with Lambda execution but no output is output to CloudWatch Logs, but are logs output when running with the Lambda test button? Is API Gateway created using HTTP API? For HTTP API, the following settings may be required.
I think you can get the error by using the "$context.integrationErrorMessage" log variable.
https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-troubleshooting-lambda.html

profile picture
EXPERT
answered 4 months ago
0

Hello.

If EFS mounting fails from Lambda, I think something will be output to CloudWatch Logs.
Was there a policy set for Lambda's IAM role to output logs to CloudWatch Logs?
https://repost.aws/knowledge-center/lambda-invoke-efs

Basically, I think you can identify the problem by looking at the logs, but if the service runs on a VPC, you also need to check the settings around the network, so in that case, use "Reachability Analyzer" to check communication.
https://repost.aws/knowledge-center/vpc-connectivity-reachability-analyzer

profile picture
EXPERT
answered 4 months ago
profile pictureAWS
EXPERT
reviewed 4 months ago
  • Yes, my Lambda was correctly configured to write to logs. Once the connectivity issue was resolved, it started writing log messages again. But while the connectivity issue persisted, no log messages were written, making it very difficult to determine the cause. I have an Api Gateway in front of the Lambda, so I turned on logging there, and could see the Api Gateway sending Lambda integration requests, but receiving the 500 error. But past that, there were no log messages available anywhere.

    Reachability Analyzer is a useful tool, but I would not have thought of it until I suspected a network issue with EFS. Suspecting an issue with the EFS connection is what took most of the time.

  • It is unclear why there is a problem with Lambda execution but no output is output to CloudWatch Logs, but are logs output when running with the Lambda test button? Is API Gateway created using HTTP API? For HTTP API, the following settings may be required. https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-troubleshooting-lambda.html

  • Good eye on noticing the logging variable for the HTTP api gateway. I believe that would give more information about the issue. I'd like to mark a comment of yours as the solution, so if you want the points, go ahead and repost your comment as a top level answer, and I'll accept it. Cheers.

0

According to @Riku_Kobayahi's insight, the ability to detect it faster may have been the "$context.integrationErrorMessage" logging variable on the Api gateway sitting in front of my Lambda. https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-troubleshooting-lambda.html

I can't test it now, but appears that that variable should output some more informative messaging about the cause of the Lambda integration error.

answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions