Lambda function times out during Init and then again during Invoke with "Error Type: Runtime.Unknown"

Question

I have a Lambda function that constantly times out with the following log:

```
2023-11-28T12:15:10.045+09:00	INIT_REPORT Init Duration: 10009.47 ms Phase: init Status: timeout
2023-11-28T12:16:10.132+09:00	INIT_REPORT Init Duration: 60060.90 ms Phase: invoke Status: error Error Type: Runtime.Unknown
```
Both the function and it's init, is in no way compute or memory heavy.  When it does run properly, it executes in less than 1 sec.  (Please check below for what I need to do to get it to execute.)

The function uses a layer, and some boilerplate for getting an AWS secrets object and connecting to Sentry + setting up logging. But this is the same boilerplate we use for all of our Lambdas and it has yet to ever cause any similar issues on any other functions.

Now, there is a way I can get the function to execute: If I remove the layer from the function once, saves, and then re-attach the layer again, the function will execute properly again. However, this only lasts until I update the function code. Once I update the function code, the function starts timing out in Init again, until I once more remove and re-attach the layer. The result is the same whether I update the code via CLI, code editor in the console, or via zip file upload in the console.

The function has 10240MB memory available, but the logs always says that it used no more than 32MB.

The runtime environment is python 3.11, x86. The layer is built for the same runtime environment.

I just can't seem to find information anywhere about what exactly goes wrong in the init phase, and I have no clue why the "resetting" of the layer temporarily helps.

Is there any way that I can debug the init phase? Does anyone have any idea what might be happening? Is there an issue with the layer?
Any help would be much appreciated.

Answer

The issue you're experiencing with Lambda timeouts during the initialization phase and unknown errors during invocation can be quite challenging to diagnose. However, I can provide some troubleshooting steps and potential areas to investigate that might help you resolve the problem.

1. Layer Configuration:
   Double-check the configuration and contents of the layer you're using. Ensure that the layer is correctly packaged and compatible with the Python 3.11 runtime environment. There might be something in the layer that's causing intermittent issues during initialization.

2. Dependency Loading and Initialization:
   During the initialization phase, Lambda loads the function code and its dependencies. Ensure that there are no issues with dependencies loading from the layer or any initialization code that might be causing delays or timeouts.

3. AWS Secrets Manager or Sentry Initialization:
   The initialization process includes fetching secrets from AWS Secrets Manager and setting up Sentry. Check if there might be occasional delays or issues with these services that could lead to timeouts during initialization.

4. Logging and Error Handling:
   Ensure that your logging mechanism and error handling within the initialization phase are robust. Any issues with logging or error handling might cause unexpected behavior or delays during initialization.

5. Memory or Resource Utilization:
   Even though your function uses a significant memory allocation (10240MB), monitor resource utilization during initialization. Check if there are any unusual spikes or constraints in memory usage that might cause timeouts.

6. Debugging and Logs:
   Enable detailed logging statements within your Lambda function's initialization phase to capture more granular information about the execution flow. Log timestamps, resource utilization, and any custom messages that might help in pinpointing the issue.

7. AWS Support:
   If the issue persists and you're unable to identify the root cause, consider reaching out to AWS support. They might have more insights or tools to diagnose and resolve this kind of issue.

Regarding debugging the initialization phase specifically, Lambda doesn't provide direct debugging capabilities during the initialization phase. However, adding detailed logging and handling exceptions meticulously within your initialization code can help in diagnosing issues.

Given that you've noticed a temporary resolution by detaching and reattaching the layer, there might be something in the layer setup or the way it's associated with the function that triggers a temporary fix. Investigating this aspect more thoroughly might provide insights into the root cause.

Ultimately, this issue might require a systematic approach, meticulous logging, and potentially trial-and-error testing to identify and resolve the issue.

Lambda function times out during Init and then again during Invoke with "Error Type: Runtime.Unknown"

Relevant content