Lambda loses ability to connect to outside web services

0

Hello,

AWS Lambda. C#. .NET Core 6.0. Written & published in Visual Studio. Project created using the latest Template from AWS. Hosts a web API. Is using a VPC since a web service I need to call has to whitelist the IP address.

The Lambda will function well for a while - sometimes even under heavy load. Then all of a sudden any calls to outside 3rd party web services start failing. 504 - Gateway timeout error. Wait a bit - starts working again. Sometimes minutes after publishing and only a few invocations. Sometimes after hundreds of invocations.

Checked all my code. Best practices all used around the HTTPClient calls. Everything is disposed of properly. Works fine running locally from Visual Studio.

Relative newbie on AWS. Tons of experience doing similar things on Azure without issue.

Any advice is very much appreciated. This has been tormenting me for days!

Thanks in advance!

2 Answers
3

The issue might be caused by cold starts in AWS Lambda, where new execution environments (containers) are created to handle incoming requests, leading to initialization delays and potential timeouts when making outbound calls to external web services. I suggested cold starts as a potential cause because the issue you described seems to happen intermittently, sometimes after only a few invocations, and sometimes after the Lambda function has been running for a while.

It would be good you read this article about Operating Lambda.

Additionally, to mitigate cold starts, you can use AWS Lambda's provisioned concurrency feature, which keeps a pool of initialized execution environments ready to handle incoming requests without the need for cold starts. Alternatively, you can implement techniques like periodic invocations or AWS Lambda Extensions to keep your functions warm and avoid cold starts.

Please note that while this solution is a good starting point for debugging, I cannot guarantee it will resolve your issue.

profile picture
EXPERT
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago
  • Hi! Thanks for taking the time time to answer. I followed the advice in the article. Fired up provisioned concurrency on the lambda (10 instances). Also tried the periodic invocations.

    Some more details - the 3rd party endpoint I am hitting results in a few hundred milliseconds both deployed (when it works) and when run locally. I get zero failures running tens of thousands of calls as quickly as I can when running locally - always getting a response in a few hundred milliseconds. When deployed the HTTPClient call fails on timeout (tried everything from 10-100 seconds) about 1/3 of the time. The 3rd party vendor says their logs don't show those ever getting to them.

    There is a VPC configured on this. I know it is working because the 3rd party library required a whitelisted IP address. Maybe that is the issue??? I have no idea how to even check.

    I am really pulling my hair out on this. Thanks for the help.

1

It is the 3rd party web service returns the 504 error, right? If so, the load on the 3rd party web service is the problem.

I would measure the processing time of the 3rd party's invocation part, check the response from the 3rd party, and contact the 3rd party with the details of the problem.

profile picture
EXPERT
shibata
answered 2 months ago
  • Hi! Thanks for taking the time time to answer. Sadly I can't blame the 3rd party on this one. It doesn't apear to even it make it out to them. Their logs show no incoming traffic at that point. And when I run the same code locally it runs flawlessly. Only when the lambda is deployed. Thanks though!

  • How do IPs be fixed to be registered in the list?

    Are you running in multiple AZs and can't communicate with the outside world only when running in a specific AZ?

    You may be able to find connection problems from VPC Flow Logs.

    https://repost.aws/knowledge-center/vpc-flow-logs-and-cloudwatch-logs-insights

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions