Hi all. We're just starting with SSM and hoping to use this quite extensively moving forward. I have a small handful of devices online using it but I just noticed that one of my devices has gone offline. Luckily it is not in the field yet so I was able to access it locally. What appears to have happened was SSM Agent was attempting to update but failed and never came back. I'm not so concerned that it didn't update and more concerned that it didn't start again. Going through the ssm agent logs I came across this line
"standardError": "E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\nWARNING: Could not install the python3-apt, this may cause the patching operation to fail.\nfailed to run commands: exit status 1
The SSM update log itself ends like this
2022-01-20 21:47:09 DEBUG UpdateInstanceInformation Response{
}
2022-01-20 21:47:09 INFO initiating cleanup of other versions in amazon-ssm-agent and amazon-ssm-agent-updater folder
2022-01-20 21:47:09 INFO removing artifacts in the folder: /var/lib/amazon/ssm/update/amazon-ssm-agent
2022-01-20 21:47:09 INFO removed files and folders: 3.1.821.0
2022-01-20 21:47:09 INFO removing artifacts in the folder: /var/lib/amazon/ssm/update/amazon-ssm-agent-updater
2022-01-20 21:47:09 INFO removed files and folders: 3.1.715.0
2022-01-20 21:47:09 INFO initiating cleanup of files in update download folder
2022-01-20 21:47:09 INFO Successfully downloaded manifest
Successfully downloaded updater version 3.1.821.0
Updating amazon-ssm-agent from 3.1.715.0 to 3.1.821.0
Successfully downloaded https://s3.us-east-2.amazonaws.com/amazon-ssm-us-east-2/amazon-ssm-agent/3.1.715.0/amazon-ssm-agent-ubuntu-amd64.tar.gz
Successfully downloaded https://s3.us-east-2.amazonaws.com/amazon-ssm-us-east-2/amazon-ssm-agent/3.1.821.0/amazon-ssm-agent-ubuntu-amd64.tar.gz
Initiating amazon-ssm-agent update to 3.1.821.0
failed to install amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 125
exit status 125
Initiating rollback amazon-ssm-agent to 3.1.715.0
failed to uninstall amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 2
exit status 2
Failed to update amazon-ssm-agent to 3.1.821.0
Then the error log gives me this
2022-01-20 16:29:54 ERROR [Submit @ processor.go.140] [ssm-agent-worker] [MessagingDeliveryService] [Association] [associationId=5752f0d0-1f57-492e-83f7-740484b81d73] Document Submission failed: Job with id 5752f0d0-1f57-492e-83f7-740484b81d73 already exists
2022-01-20 21:47:02 ERROR [AppendError @ context.go.129] failed to install amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 125
exit status 125
Could anyone help me out here? I cannot have these fail like this when they go out into the wild.
The OS is Ubuntu Server 18 on this box, 20 on others we have.