How do I resolve "failed to run commands: exit status 247" error when I run AWS-RunPatchBaseline on a Linux instance?

3 minute read
1

When I use an AWS Systems Manager (SSM) to patch my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance, I get a "failed to run commands: exit status 247" error.

Short description

The words "exit status 247" means that there was an error when you run a command like AWS-RunPatchBaseline. For example, an Amazon EC2 instance might generate this error when the instance doesn't have enough memory space (RAM) to run the patch. However, the error message doesn't provide a cause for this error.

Resolution

Identify the reason for the command's failure

To troubleshoot any patching failures, review the contents of the stdout and stderr files. You can locate the files in the following location on your Linux managed node.

/var/lib/amazon/ssm/<instance-id>/document/orchestration/<Run-Command-execution-id>/awsrunShellScript/PatchLinux

If you don't see an anomaly in stdout, and if stderr reflects "failed to run commands: exit status 247", then check the system logs. Exhausted memory is the likely cause.

Use the logs to verify the cause of error

Follow the options listed below to verify that the logs show Out of memory: kill process trails.

Check the instance console output

Use the instance console output to view the system logs.

Use the operating system's native commands

Use a command such as journalctl or dmesg from the OS terminal.

To check the logs in reverse order, run the journalctl command with the -r option:

# sudo journalctl -r

Example output:

Jul 11 12:02:11 ip-10-0-5-170.ec2.internal kernel: Out of memory: Killed process 5598 (python2.7) total-vm:713908kB, anon-rss:288688kB, file-rss:2184kB, shmem-rss:0kB, UID:0 pgtables:1080kB oom_score_ad
Jul 11 12:02:11 ip-10-0-5-170.ec2.internal kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python2.7,pid=5598,uid=0

-or-
Run the dmesg command with the -T option:

# sudo dmesg -T

Example output:

[Thu Jul 11 12:02:13 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python2.7,pid=5598,uid=0
[Thu Jul 11 12:02:13 2024] Out of memory: Killed process 5598 (python2.7) total-vm:713908kB, anon-rss:288688kB, file-rss:2184kB, shmem-rss:0kB, UID:0 pgtables:1080kB oom_score_adj:0

In the preceding examples, the SSM Patch Manager used Python 2.7.

The stdout output below shows that the Python's process ended soon after it began, as you can see in the timestamps. This result suggests that the instance didn't have enough memory to run the patch operation.

# head /var/lib/amazon/ssm/<instance-id>/document/orchestration/238a1da0-6ed7-46f5-8d75-87363d18e160/awsrunShellScript/PatchLinux/stdout
/usr/bin/python3
/usr/bin/python2.7
/usr/bin/python2
/usr/bin/python
/usr/bin/yum
Using Yum version: 3.4.3
Using python binary: 'python2.7'
Using Python Version: Python 2.7.18
07/11/2024 11:58:09 root [INFO]: Downloading payload from https://s3.us-east-1.amazonaws.com/aws-ssm-us-east-1/patchbaselineoperations/linux/payloads/patch-baseline-operations-1.132.tar.gz
07/11/2024 11:58:10 root [INFO]: Attempting to import entrance file os_selector

Change the instance type

To run the patch baseline on the instance, change the instance type to increase the instance's memory. Or, investigate the instance's current memory usage to determine if the instance is using a high amount of resources or if the instance has other anomalies. After you discover the issues, update the instance as needed.

Related information

How do I troubleshoot Systems Manager Run Command failures?

Why is my EC2 Linux instance becoming unresponsive due to over-utilization of resources?

Capture a screenshot of an unreachable instance

How do I troubleshoot a failed Patch Manager (Linux) operation?

AWS OFFICIAL
AWS OFFICIALUpdated 2 months ago