How to debug CLI crash when running Lightsail container commands?

0

I only use a couple of commands in my workflow but very often both will randomly throw a cryptic error in the middle of a deploy on CI:

  • aws lightsail get-container-service-deployments --region us-west-2 --service-name my_service --output json
  • aws lightsail get-container-images --region us-west-2 --output json --service-name my_service

My CI stack is Ubuntu Server 18.04 which runs a Github action-runner service.

Signal received: -1695824320, errno: 32575

Stack trace:

/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(aws_backtrace_print+0x4d) [0x7f3f9483edbd]
/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(+0x68513) [0x7f3f947b5513]
/lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f3f9af46040]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1f9ed0) [0x7f3f9ab65ed0]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0xbb58b) [0x7f3f9aa2758b]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1fa930) [0x7f3f9ab66930]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(PyGC_Collect+0x81) [0x7f3f9ab67aa1]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_FinalizeEx+0xe2) [0x7f3f9ab3ff02]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_Exit+0x8) [0x7f3f9ab40818]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1d8c8b) [0x7f3f9ab44c8b]
aws(+0x378b) [0x55705f60f78b]
aws(+0x3b1f) [0x55705f60fb1f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f3f9af28bf7]
aws(+0x24fa) [0x55705f60e4fa]

Any ideas what to do about this?

edit Jan 1, 2022

I turned on the --debug flag. There is no debug output. The command seems to immediately crash. I find if I run these command multiple times, they randomly crash with and sometimes it prints "Segmentation fault" which looks to likely to be a memory access/management issue :(

Something else worth saying is this issue only presents itself on Ubuntu distros. I ran out of options to try and reinstalled other OS like ArchLinux and did not experience the same problem. Unfortunately some of the other software I use isn not well supported under ArchLinux and I have to come back to Ubuntu :(

2개 답변
0

One of the first things you can do when the AWS CLI produces results that you don't expect, is get more detail by running the command again and including the --debug option at the end of the command line. This causes the AWS CLI to report details about every step it takes to process your command, send the request to the AWS servers, receive the response, and process the response into the output you see.

답변함 3년 전
  • I did try the --debug flag in my deploy automation script. The problem is I need to capture the JSON output and process it. Getting the cli to output its debug information means debug output will be mixed with the JSON I'm trying to parse; so the whole process won't work. I haven't worked out how to both turn on debug output and still be able to process the command output.

    Scratch that. --debug flag sends its output to stderr. I must have been confused by something else.

0

Hi! Along with what other users have suggested: using the debug option at the end of the command line commands can help.

As for help troubleshooting the lightsail deploy, the following additional API calls may help:

If there's enough progress, you can also look at the system status check or the instance status check for more information: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-viewing-instance-health-metrics#viewing-instance-metrics-console

You may also want to try bumping up the lightsail server to one with more memory, CPU, space, etc depending on what your error is and see if that helps.

답변함 3년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠