Amazon Linux 2023, kernel 6.1.77-99.164.amzn2023.x86_64, running on Lightsail
in ap-south-1.
Three times in five months, systemd-networkd has failed to install a route on
ens5 and marked the interface Failed without ever retrying. The instance loses
all outbound connectivity and only recovers on reboot. Total downtime ~4.5 hours.
Identical log signature each time:
systemd-networkd[PID]: ens5: Could not set route: Connection timed out
systemd-networkd[PID]: ens5: Failed
No further systemd-networkd entries until shutdown.
Occurrences (UTC):
2026-03-24 03:47:08 -> 60 min outage
2026-07-25 16:30:56 -> 107 min outage
2026-08-18 06:59:44 -> 105 min outage
Evidence from the most recent occurrence:
-
Outbound transmit dropped to exactly zero while receive continued (sar -n DEV):
07:50 ens5 rxpck/s 8.52 txpck/s 6.97
08:00 ens5 rxpck/s 7.81 txpck/s 0.00
08:40 ens5 rxpck/s 2.37 txpck/s 0.00
-
Zero interface errors (sar -n EDEV): rxerr, txerr, rxdrop, txdrop, txcarr
all 0.00 throughout. No hardware fault.
-
Kernel logged nothing at the failure timestamps - journalctl -k for those
windows returns "No entries".
-
refresh-policy-routes@ens5.service then failed every 2 minutes with
"Timeout occurred while waiting for network connectivity" until reboot.
-
IMDS (169.254.169.254) and VPC DNS both became unreachable from inside the
instance.
Ruled out: memory (no OOM events), disk (18% used, 1% inodes), CPU steal
(max 0.11%, idle above 99.3%), hardware (system status check passed, only the
instance status check failed).
For comparison, a separate Ubuntu 24.04 Lightsail instance in the same region
with a similar 1GB spec has 350 days uptime and zero occurrences. Its network
configuration is a static netplan file written once by cloud-init, whereas
AL2023 uses 80-ec2.network with ec2-net-utils periodically reconfiguring the
interface.
Questions:
-
Is this a known issue with the systemd version shipped in AL2023, or with
ec2-net-utils on Lightsail?
-
Is there a supported way to make systemd-networkd retry after a route
installation failure, rather than leaving the interface in Failed state
indefinitely?
