Why is the /var/log directory missing logs in my EC2 instance that runs on AL2023?

Resolution

The rsyslog service maintains various log files in the /var/log directory. Amazon Linux 2 (AL2) keeps this service for backward compatibility. However, the rsyslog service isn't installed in AL2023 instances by default. As a result, the corresponding files in the /var/log directory, such as /var/log/messages, aren't available in AL2023.

The default service manager systemd in AL2 and AL2023 uses systemd-journald to create logs.

Query systemd-journald

Unlike rsyslog, the systemd-journald service doesn't record information into files, such as messages, secure, maillog, and spooler. Instead, systemd-journald stores system information in /var/log/journal.

To query the journal, you must run the journalctl command.

To check all logs, run the following command:

journalctl

To check logs in reverse order, run the following command:

journalctl -r

To check the logs for a specific time duration, run the following command:

journalctl --since -10min

Note: Replace 10 with the number of minutes for your time duration.

To check logs within a specific timestamp, run the following command:

journalctl --since "2023-06-21 10:50:00" --until "2023-06-21 11:00:00"

Note: Replace 2023-06-21 10:50:00 with the beginning date and time of your time stamp and 2023-06-21 11:00:00 with the end.

To check logs that are specific to a service, run the following command:

journalctl -u sshd

Note: The preceding command checks the SSH service. If needed, then replace sshd with your service.

To check logs based on log level of messages, run the following command:

journalctl -p err

Note: The preceding command checks the err log level. Replace err with another log level, such as emerg, alert, or crit, to filter the output by message priorities.

Activate log files in the /var/log/ directory

To install the rsyslog package on AL2023 and start the service, run the following commands:

# dnf install rsyslog
# systemctl enable rsyslog --now

To validate that Amazon EC2 writes the logs to the messages files, run the following command to add a test line and check /var/log/messages:

# logger test_line  
# grep "test_line" /var/log/messages

Example output:

Jun 22 08:06:08 localhost root[1771]: test_line