Have a lambda function that is throwing an error of "No space left on device". The lambda function creates a custom resource handler defined within the lambda python code:
response = cfn.register_type(
Type='RESOURCE',
TypeName='AWSQS:MYCUSTOM::Manager',
SchemaHandlerPackage="s3://xxx/yyy/awsqs-mycustom-manager.zip",
LoggingConfig={"LogRoleArn": "xxx", "LogGroupName": "awsqs-mycustom-manager-logs"},
ExecutionRoleArn="xxx"
The lambda function when created has the following limits set: 4GB of Memory and 4GB of Ephemeral space.
However, I was still receiving a no space on device even thought the '/tmp/' is specified and this is plenty of space. Doing additional digging I added a "df" output inside of the code/zip file. When the output prints is shows that only 512MB of space is available in temp?
Filesystem 1K-blocks Used Available Use% Mounted on
/mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 27190048 22513108 3293604 88% /
/dev/vdb 1490800 14096 1460320 1% /dev
/dev/vdd 538424 872 525716 1% /tmp
/dev/root 10190100 552472 9621244 6% /var/rapid
/dev/vdc 37120 37120 0 100% /var/task
Its like a new instance was created internally and did not adopt the size from the parent. Forgive me if technically my language is incorrect as this is the first time busting this out and seeing this type of error.
Just has me confused as too what is going on under the covers, and I can find no documentation on how to increase the ephemeral storage within the handler even though the originating lamda function in which this is defined has already had the limits increased.
AWS 官方已更新 2 年前
it happens each and every time. The invocation of a DF in the originating lambda shows exactly as you have described, yet when the code is invoked inside of the handler it shows a different space size for /tmp. That is what is throwing me off. Like its a new boto instance created within the lambda handler. Again forgive me if my technical language is a challenge.
What is your function doing? Can you share the code?
What do you mean by "originating lambda"? How many functions do you have?
I have a few functions but when I modify the original lamda code with the following:
def register(cfn): response = cfn.register_type( Type='RESOURCE', TypeName='AWSQS:MYCUSTOM::Manager', SchemaHandlerPackage="s3://xxx/yyy/awsqs-mycustom-manager.zip", LoggingConfig={"LogRoleArn": "xxx", "LogGroupName": "awsqs-mycustom-manager-logs"}, ExecutionRoleArn="xxx" ) dfout = os.popen('df') print( dfout.read())
I can view the cloudwatch logs and see the output for the lambda:
/mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 8351792 6720731 1626965 81% / /dev/vdd 4229728 18152 4195192 1% /tmp
Which is 4GB and good.
However, when the awsqs-mycustom-manager.zip is updated with the exact same debug:
Filesystem 1K-blocks Used Available Use% Mounted on /mnt/root-rw/opt/amazon/asc/worker/tasks/rtfs/python3.7-amzn-201803 8351792 6720731 1626965 81% / /dev/vdb 1490800 14092 1460324 1% /dev /dev/vdd 538424 876 525712 1% /tmp /dev/root 10190100 552464 9621252 6% /var/rapid /dev/vdc 37120 37120 0 100% /var/task
returns 512MB ...
If I understand correctly, awsqs-mycustom-manager is a new function. As a new function you need to change its configuration as well. It does not inherit the configuration from this function.