AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
UserData script being killed on AL3 AMI
I have a UserData script for an EC2 instance running on the latest AL3 AMI. This has been in place for years and has been working fine up until recently (last couple of months?). My script basically installs Ansible, downloads a playbook, and runs the playbook. Lately, the script has been randomly failing with a mysterious "1620 killed" message.
Cloud-init v. 22.2.2 running 'modules:config' at Thu, 30 Jul 2026 14:49:28 +0000. Up 6.31 seconds.
Cloud-init v. 22.2.2 running 'modules:final' at Thu, 30 Jul 2026 14:49:29 +0000. Up 6.90 seconds.
+ dnf install ansible pip -y
Amazon Linux 2023 repository 73 MB/s | 72 MB 00:00
/var/lib/cloud/instance/scripts/part-001: line 3: 1620 Killed dnf install ansible pip -y
+ pip install boto3
/var/lib/cloud/instance/scripts/part-001: line 4: pip: command not found
This EC2 instance is in an AutoScaling group and gets replaced every morning. When the error happens I have force the instance to die so it gets replaced. Sometimes the next instance works fine but sometimes it has to kill/create 5-10 instances before one works properly.
There doesn't seem to be any rhyme or reason to this error, and it seems to be happening more and more frequently. I'm using the generic AL3 AMI published by AWS and not doing anything weird or unusual. In fact, the "dnf install ansible pip -y" command is one of the first commands.
Did something change recently with the AL3 AMIs?
- Topics
- Compute
- Language
- English
asked a month ago91 views
- Newest
- Most votes
- Most comments
The re:Post agent is correct about the OOM killer, but missed why this is happening now. As AL2023 receives ongoing updates, the DNF repository metadata grows significantly. Parsing this enlarged metadata causes temporary RAM spikes that kill processes on smaller instances.
You don't need to pay for a larger instance type. The standard, cost-free fix is to create a temporary swap file in your UserData just before running dnf:
fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
dnf install ansible pip -y
# Optional cleanup
swapoff /swapfile
rm -f /swapfile
The 'Killed' message you're seeing typically indicates that a process was terminated by the Linux Out of Memory (OOM) killer. This happens when the system runs out of available memory and the kernel forcibly terminates processes to free up resources. Given that your dnf install command is being killed early in the UserData script execution, this strongly suggests your instance type doesn't have enough memory to complete the package installation.
A few things to consider:
First, check your instance type's memory allocation. The dnf package manager, especially when installing multiple packages with dependencies, can consume significant memory during the installation process. If you're using a smaller instance type (like t2.micro or t3.micro with only 1GB of RAM), you may be hitting memory limits.
Second, the intermittent nature of the failures could be explained by varying system memory usage at launch time. Different background processes or services starting up can affect available memory, which is why sometimes it works and sometimes it doesn't.
To troubleshoot this, you can check the cloud-init output log at /var/log/cloud-init-output.log after connecting to a failed instance. You should also examine the system logs (dmesg or /var/log/messages) for OOM killer messages that would confirm memory exhaustion.
For a solution, consider upgrading to an instance type with more memory, or modify your UserData script to install packages sequentially rather than all at once. You could also add error handling and retry logic to make the script more resilient to transient failures.
Regarding recent changes to AL3 AMIs - while AMI updates do occur regularly and could potentially include changes to default memory usage patterns or package sizes, the OOM behavior you're describing is more likely related to the resource constraints of your instance type rather than a specific AMI change.
Sources
Run commands when you launch an EC2 instance with user data input - Amazon Elastic Compute Cloud
Understand why EC2 instance stopped | AWS re:Post
Relevant content
asked 2 years ago
If a t4g.nano EC2 instance can't even install a single package when it's freshly booted, then I'm not sure what you might use it for at all. The
ansiblepackage is not that big.