Lambda Handler No Space On Device Error

0

Have a lambda function that is throwing an error of "No space left on device". The lambda function creates a custom resource handler defined within the lambda python code:

response = cfn.register_type( Type='RESOURCE', TypeName='AWSQS:MYCUSTOM::Manager', SchemaHandlerPackage="s3://xxx/yyy/awsqs-mycustom-manager.zip", LoggingConfig={"LogRoleArn": "xxx", "LogGroupName": "awsqs-mycustom-manager-logs"}, ExecutionRoleArn="xxx"

The lambda function when created has the following limits set: 4GB of Memory and 4GB of Ephemeral space.

However, I was still receiving a no space on device even thought the '/tmp/' is specified and this is plenty of space. Doing additional digging I added a "df" output inside of the code/zip file. When the output prints is shows that only 512MB of space is available in temp?

Filesystem 1K-blocks Used Available Use% Mounted on /mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 27190048 22513108 3293604 88% / /dev/vdb 1490800 14096 1460320 1% /dev /dev/vdd 538424 872 525716 1% /tmp /dev/root 10190100 552472 9621244 6% /var/rapid /dev/vdc 37120 37120 0 100% /var/task

Its like a new instance was created internally and did not adopt the size from the parent. Forgive me if technically my language is incorrect as this is the first time busting this out and seeing this type of error.

Just has me confused as too what is going on under the covers, and I can find no documentation on how to increase the ephemeral storage within the handler even though the originating lamda function in which this is defined has already had the limits increased.

asked 2 years ago1810 views
1 Answer
0

Does it happen on each invocation or only after a few of them? the /tmp remains between invocations in the same sandbox. If it happens in each invocation, need to further investigate. If it only happens after a few invocations, make sure to delete the files that were created in the previous invocations.

BTW, I tried df and it worked just fine:

Filesystem                          1K-blocks    Used Available Use% Mounted on
/dev/vdd                             4130944   17736   4096824   1% /tmp

Maybe there is something wrong in the way you configured it?

profile pictureAWS
EXPERT
Uri
answered 2 years ago
  • it happens each and every time. The invocation of a DF in the originating lambda shows exactly as you have described, yet when the code is invoked inside of the handler it shows a different space size for /tmp. That is what is throwing me off. Like its a new boto instance created within the lambda handler. Again forgive me if my technical language is a challenge.

  • What is your function doing? Can you share the code?

  • What do you mean by "originating lambda"? How many functions do you have?

  • I have a few functions but when I modify the original lamda code with the following:

    def register(cfn): response = cfn.register_type( Type='RESOURCE', TypeName='AWSQS:MYCUSTOM::Manager', SchemaHandlerPackage="s3://xxx/yyy/awsqs-mycustom-manager.zip", LoggingConfig={"LogRoleArn": "xxx", "LogGroupName": "awsqs-mycustom-manager-logs"}, ExecutionRoleArn="xxx" ) dfout = os.popen('df') print( dfout.read())

    I can view the cloudwatch logs and see the output for the lambda:

    /mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 8351792 6720731 1626965 81% / /dev/vdd 4229728 18152 4195192 1% /tmp

    Which is 4GB and good.

    However, when the awsqs-mycustom-manager.zip is updated with the exact same debug:

    Filesystem 1K-blocks Used Available Use% Mounted on /mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 8351792 6720731 1626965 81% / /dev/vdb 1490800 14092 1460324 1% /dev /dev/vdd 538424 876 525712 1% /tmp /dev/root 10190100 552464 9621252 6% /var/rapid /dev/vdc 37120 37120 0 100% /var/task

    returns 512MB ...

  • If I understand correctly, awsqs-mycustom-manager is a new function. As a new function you need to change its configuration as well. It does not inherit the configuration from this function.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions