How to monitor lambda functions running inside Greengrass V2 core device

Question

I am trying to understand a bit more on how lambda functions run on Greengrass.

This question is more about debugging the execution of a lambda function deployed in a Raspberry Pi 4, running Raspbian.

I deployed a Lambda component to my Greengrass core device. However, even though the logs say that the function is running (logs below), I cannot really understand or see if it is actually doing something.

2023-05-05T06:05:27.275Z [INFO] (pool-2-thread-20) com.example.LambdaFunction: shell-runner-start. {scriptName=services.com.example.LambdaFunction.lifecycle.startup.script, serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=STARTING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-05-05T06:05:27.617Z [INFO] (Copier) com.example.LambdaFunction: stdout. Started process: 1075. {scriptName=services.com.example.LambdaFunction.lifecycle.startup.script, serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=STARTING}
2023-05-05T06:05:27.622Z [INFO] (Copier) com.example.LambdaFunction: Startup script exited. {exitCode=0, serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=STARTING}
2023-05-05T06:05:28.077Z [INFO] (pool-2-thread-21) com.example.LambdaFunction: lambda_runtime.py:402,Status thread started. {serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=RUNNING}
2023-05-05T06:05:28.085Z [INFO] (pool-2-thread-21) com.example.LambdaFunction: lambda_runtime.py:154,Running [arn:aws:lambda:eu-central-1:109139691401:function:gg-lambda-test:1]. {serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=RUNNING}

This is the logic for my function: https://github.com/aws-samples/aws-greengrass-samples/blob/master/hello-world-counter-python/greengrassHelloWorldCounter.py

It is essentially just counting the number of instances that a message has been published into the topic.

For testing this lambda function, I created a subscription and a topic. The function should be invoked ideally when there is a new message in the topic.

I followed this tutorial https://docs.aws.amazon.com/greengrass/v1/developerguide/long-testing.html

However, whenever I try to publish something into a topic, say /hello/world/counter. Instead of getting the message from the lambda function code:
'message': 'Hello world! Sent from Greengrass Core.  Invocation Count: {}'.format(my_counter)

I get the same message that I put in the MQTT test AWS console: 
{
  "message": "Hello from AWS IoT console"
}

I suspect that the function is not being invoked at all, but how is it possible if it is actually running?

I have been stuck on this for a few days and would love to do some progress for my master thesis.

Any help would be appreciated,

Best,
Ed

Answer

Hi Ed
Thanks for using Greengrass and thanks for reaching out!

I understand you were trying to follow along the Long Testing guide in our tutorials doc (https://docs.aws.amazon.com/greengrass/v1/developerguide/long-testing.html). So firstly, all Greengrass log entries include a timestamp, log level, and information about the event. For example

2023-05-05T06:05:27.275Z [INFO] (pool-2-thread-20) com.example.LambdaFunction: shell-runner-start. {scriptName=services.com.example.LambdaFunction.lifecycle.startup.script, serviceInstance=0, serviceName=com.example.LambdaFunction, currentState=STARTING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}

contains the timestamp, log level: INFO and event related info, ie, this was logged when Greengrass was executing the startup script of the component "com.example.LambdaFunction." You can use logs to monitor events and troubleshoot issues, for example, you can log a friendly message in the lambda function handler before it publishes anything to verify that it is actually being invoked.

Coming to the actual issue, the long testing guide specifies after 5 that "This single subscription goes in one direction only: from the Greengrass_HelloWorld_Counter Lambda function to AWS IoT. To invoke (or trigger) this Lambda function from the cloud, you must create a subscription in the opposite direction."
If you created only one topic and both the lambda function and the IoT console are subscribing to this topic then any message you publish on this topic, the same message will show up on the IoT console again.

Hope this helps!

How to monitor lambda functions running inside Greengrass V2 core device

Relevant content