I want to troubleshoot my Amazon Elastic Compute Cloud (Amazon EC2) Linux RHEL 7 or RHEL 8 instance bootstrap. I want to log the user-data invocation and then ship it to the console logs.
Short description
To troubleshoot issues on your EC2 instance bootstrap, add code to your user-data bash script that redirects all output to /var/log/user-data.log and /dev/console. You can do this even if you can't access the instance through SSH. After you run the code, your user-data invocation logs appear in your console.
Note: This resolution is for RHEL 7 and RHEL 8 only. For information on Amazon Linux and Amazon Linux 2, see How can I send user-data output to the console logs on an EC2 instance running Amazon Linux or Amazon Linux 2?
Resolution
-
Open the Amazon EC2 console.
-
Launch a new instance and SSH into it.
-
Edit the GRUB_CMDLINE_LINUX line in /etc/default/grub and change "console=ttyS0,115200n8 console=tty0" to "console=tty1 console=ttyS0":
RHEL 7
# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"
RHEL 8
# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
-
Recreate the /boot/grub2/grub.cfg file:
RHEL 7
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1062.1.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1062.1.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-026767dbe06a4910a5ce3bd0def903c0
Found initrd image: /boot/initramfs-0-rescue-026767dbe06a4910a5ce3bd0def903c0.img
done
RHEL 8
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done
-
Stop the instance.
-
Create a new AMI from the modified instance.
-
Launch a new instance from the new AMI.
-
To redirect the user-data output console, run the following command:
#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
yum -y update
echo "Hello from user-data!"
The following line redirects the user-data output:
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
You receive a console output that's similar to the following example:
RHEL 7:
subscription-manager.x86_64 0:1.24.26-3.el7_8
subscription-manager-rhsm.x86_64 0:1.24.26-3.el7_8
subscription-manager-rhsm-certificates.x86_64 0:1.24.26-3.el7_8
sudo.x86_64 0:1.8.23-9.el7
systemd.x86_64 0:219-73.el7_8.8
systemd-libs.x86_64 0:219-73.el7_8.8
systemd-sysv.x86_64 0:219-73.el7_8.8
teamd.x86_64 0:1.29-1.el7
tuned.noarch 0:2.11.0-8.el7
tzdata.noarch 0:2020a-1.el7
util-linux.x86_64 0:2.23.2-63.el7
yum.noarch 0:3.4.3-167.el7
yum-utils.noarch 0:1.1.31-54.el7_8
Replaced:
iwl7265-firmware.noarch 0:22.0.7.0-72.el7
Complete!
+ echo 'Hello from user-data!'
Hello from user-data!
RHEL 8:
Installed:
grub2-tools-efi-1:2.02-82.el8_2.1.x86_64
kernel-4.18.0-193.13.2.el8_2.x86_64
kernel-core-4.18.0-193.13.2.el8_2.x86_64
kernel-modules-4.18.0-193.13.2.el8_2.x86_64
linux-firmware-20191202-97.gite8a0f4c9.el8.noarch
Complete!
+
echo 'Hello from user-data!'
Hello from user-data!
Note: Because your user-data is visible in the console, be sure that you don't include confidential information in the data that you send.
Related information
Run commands on your Linux instance at launch