How do I troubleshoot issues with multiple network interfaces and secondary IP addresses in AL2023?

Resolution

Update your configuration

AL2023 uses systemd-networkd instead of dhclient. If you migrate from AL2 to AL2023, then you must update how you route multiple network interfaces and secondary IP address.

Update outdated services and configuration paths

AL2023 uses different services and configuration paths to manage network interfaces and routing. Files and scripts that use outdated paths don't work in AL2023 instances.

Use the following table to understand what components in your configuration to update:

Component	AL2	AL2023
Network daemon	dhclient	systemd-networkd
Interface name	eth0, eth1, eth2	ens5, ens6, ens7 (predictable names)
Routing for multiple network interfaces	Automatic through ec2-net-utils (ifcfg/route files)	Automatic through the policy-routes service (systemd drop-ins)
Policy rule priority	Counts down from 32766, such as 32765, then 32764	Matches the table number, such as 10001, 10002
Secondary IP addresses on the primary network interface	Automatically added on the Dynamic Host Configuration Protocol (DHCP) lease renewal within about 30 minutes	Automatically added through a dedicated Instance Metadata Service (IMDS) polling timer within about 60 seconds
Conf file location	/etc/sysconfig/network-scripts/ifcfg-eth1, route-eth1	/run/systemd/network/70-ens6.network.d/
Custom routes	route-eth1 file	Drop-in CONF files in /etc/systemd/network/70-ens6.network.d/
Detach cleanup	Config files remain on disk after you detach	Config files remain in /run/ after you detach

Remove unnecessary features

Remove the following AL2 features that you no longer require in AL2023:

• /etc/sysconfig/network-scripts/ifcfg-*
• /etc/sysconfig/network-scripts/route-eth1 with default routes to specific network interfaces
• Manual ip addr add for secondary IP addresses
  Note: AL2023 polls IMDS about every 60 seconds to detect changes.
• Custom scripts in /sbin/ifup-local or udev rules for network interface setup
  Note: Instead, use policy-routes@.service.
• Manual ip rule add commands in rc.local
  Note: AL2023 automatically creates these rules.

Manually configure AL2023 features

Manually configure the following features:

• Custom destination-based routes, such as route traffic to a specific IP address through a specific network interface
• Address Resolution Protocol (ARP) settings for multiple network interfaces in the same subnet
• The default network interface for outbound traffic
• Network interfaces that you don't want to allow to have a default route

Update your file, and then apply your changes

AL2023 automatically updates your files with equivalent values. However, if you use a custom route, then you must manually translate it.

Example AL2 file:

# AL2: /etc/sysconfig/network-scripts/route-eth1 
default via 10.0.2.1 dev eth1 table 10001 
10.0.2.0/24 dev eth1 scope link table 10001 
# AL2: policy rule (created programmatically by ec2-net-utils, no rule-eth1 file) 
# Visible via: ip rule show 
# 32765: from 10.0.2.20 lookup 10001 

Example updated AL2023 file:

# AL2023: /etc/systemd/network/70-ens6.network.d/custom-routes.conf 
[Route] 
Destination=172.16.0.0/12 
Gateway=10.0.2.1 
Table=10001 
[RoutingPolicyRule] 
To=172.16.0.0/12 
Table=10001 
Priority=100 

After you create your updated file, run the following command to apply your changes:

sudo networkctl reload

If you updated your configuration and still encounter issues, then take the following troubleshooting actions based on the issue that you encounter.

Troubleshoot secondary IP addresses that don't appear on the interface

It might take up to 65 seconds for assigned secondary IP addresses to appear. The refresh-policy-routes timer frequently queries IMDS to detect new IP addresses.

To make sure that the timer is running, run the following command:

systemctl status refresh-policy-routes@ens5.timer

If the timer isn't active, then run the following command to restart it:

systemctl restart refresh-policy-routes@ens5.timer

If the timer doesn't start, then run the following command to check whether systemd-networkd is running:

systemctl status systemd-networkd

If systemd-networkd isn't running, then run the following command to start it:

systemctl restart systemd-networkd

If systemd-networkd doesn't start, then run the following command to check the logs for errors:

journalctl -u systemd-networkd --no-pager

To force AL2023 to immediately detect a secondary IP address, run the following command:

sudo systemctl start refresh-policy-routes@ens5.service

If the IP addresses still don't appear, then run the following command to check the logs for errors such as IMDS connection failures or interface configuration issues:

journalctl -u refresh-policy-routes@ens5.service --no-pager -n 20

Run the following command to make sure that IMDS returns the IP address:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
MAC=$(cat /sys/class/net/ens5/address)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/local-ipv4s

Note: If the secondary IP address doesn't appear in the IMDS output, then confirm that you assigned the IP address to the instance in the Amazon EC2 console. IMDS returns IP addresses only if they're assigned to the instance at the AWS level.

Troubleshoot secondary IP addresses that can't reach the internet

Run the following command to make sure that the operating system (OS) routing works as expected:

ip route get 8.8.8.8 from 10.0.1.11
 #   8.8.8.8 from 10.0.1.11 via 10.0.1.1 dev ens5

Note: Replace 10.0.1.11 with your secondary IP address.

If the command doesn't return the expected route, then run the following command to check the policy routing rules for your interface:

ip rule show

In the output, look for a rule that matches your source IP address. If that rule is missing, then run the following command to restart systemd-networkd and regenerate the policy routes:

systemctl restart systemd-networkd

If you use private secondary IP addresses, then they must have an associated elastic IP address. Or, their subnet must have a 0.0.0.0/0 route to a NAT gateway.

Troubleshoot issues where you can't reach the secondary network interface's IP address from another host

First, run the following command to confirm that the policy rule for the network interface's rule exists:

ip rule show | grep 10.0.2.20

Note: Replace 10.0.2.20 with the IP address of your secondary network interface.

If the rule doesn't exist, then run the following command to restart systemd-networkd to regenerate the policy routes:

systemctl restart systemd-networkd

Then, run the following command to confirm that the network interface's routing table has a default route:

ip route show table 10001

Note: Replace 10001 with the routing table number for your interface. To find your table number, run the following command:

ip rule show

To configure private access from an instance to the ens6 private IP address, check the following settings:

• Make sure that the security group on ens6 allows inbound traffic on the required port from the source security group or CIDR.
• Make sure that the network access control list (network ACL) on the subnet allows inbound traffic and outbound traffic on the ephemeral ports.
• Make sure that the source instance is in the same virtual private cloud (VPC). Or, configure VPC peering or connect attachment in AWS Transit Gateway with the required routes.

To configure public access from the internet to the ens6 public address, check the following settings:

• Make sure that ens6 has a public IP address or an associated Elastic IP address.
• Make sure that the security group on ens6 allows inbound traffic on the required port.
• Make sure that the subnet is public with a 0.0.0.0/0 route to an internet gateway in its route table.

Troubleshoot traffic that exits through the wrong interface

Important: The following resolution steps modify routing rules and routes in production and might cause your session to drop. Prepare an alternate connection method. It's a best practice to use EC2 Serial Console, as it doesn't depend on the instance's networking.

If replies to inbound connections on ens6 leave through ens5 as asymmetric routing, then there's a missing policy rule.

To resolve this issue, complete the following steps:

1. Run the following command to check whether the rule exists:

   ip rule show | grep 10.0.2.20

   Example output:

   from 10.0.2.20 lookup 10001

2. Run the following command to check the route resolution:

   ip route get 8.8.8.8 from 10.0.2.20

   Example output:

   dev ens6 table 10001

   Note: If the output shows dev ens5, then the issue is asymmetric routing.

3. Run the following command to restart the policy-routes service and rebuild the rules:

   sudo systemctl restart policy-routes@ens6.service

4. If the rule still doesn't appear, then run the following command to check the logs for errors:

   journalctl -u policy-routes@ens6.service --no-pager -n 10

5. If the rule still isn't added, then run the following commands to manually add the rule:

   sudo ip rule add from 10.0.2.20/32 lookup 10001 priority 10001
   sudo ip route add default via 10.0.2.1 dev ens6 table 10001

   Note: Replace 10.0.2.20 with your secondary IP address and 10.0.2.1 with your subnet's gateway address. The gateway address is typically the first IP address in your subnet's CIDR.

6. To make manual updates persistent, run the following command to create a drop-in file:

   sudo mkdir -p /etc/systemd/network/70-ens6.network.d/
   cat << 'EOF' | sudo tee /etc/systemd/network/70-ens6.network.d/fix-routing.conf
   [RoutingPolicyRule]
   From=10.0.2.20/32
   Table=10001
   Priority=10001
   
   [Route]
   Gateway=10.0.2.1
   Table=10001
   EOF
   sudo networkctl reload

Troubleshoot hot attach network interfaces with no connectivity

If you attach a network interface to a running instance (hot attach), then policy-routes@.service automatically configures the network interface. Newly attached network interfaces take about 30 seconds to connect.

If you still can't connect after that time, then run the following command to check whether the service ran:

systemctl status policy-routes@ens6.service

If the service didn't run, then run the following command to start it manually:

systemctl start policy-routes@ens6.service

If the service failed, then run the following command to check the logs for errors:

journalctl -u policy-routes@ens6.service --no-pager

Also, run the following command to confirm that IMDS is reachable:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/

If the preceding command returns no output or times out, then make sure that you have the following settings:

• The instance's metadata options have IMDSv2 activated.
• No iptables rules block access to 169.254.169.254.
• The instance has a route to 169.254.169.254.

Run the following command to manually update the configuration:

sudo systemctl restart policy-routes@ens6.service

Or, run the following command to reconfigure the interface:

sudo networkctl reconfigure ens6

Troubleshoot routes that disappear after reboot or networkd restart

If you manually add a route with ip route add, then the route disappears when you restart your system. To make routes persistent, place them in /etc/systemd/network. The drop-in directory name must exactly match the network file name.

To check the name configuration, run the following command:

networkctl status ens6 | grep "Network File"

If the output shows, /run/systemd/network/70-ens6.network, then your drop-in file must be at /etc/systemd/network/70-ens6.network.d/.

Run the following command to create the drop-in file:

sudo mkdir -p /etc/systemd/network/70-ens6.network.d/
cat << 'EOF' | sudo tee /etc/systemd/network/70-ens6.network.d/custom-routes.conf
[Route]
Destination=172.16.0.0/12
Gateway=10.0.2.1
Table=10001
[RoutingPolicyRule]
To=172.16.0.0/12
Table=10001
Priority=100
EOF
sudo networkctl reload

Troubleshoot issues with network interfaces in the same subnet

If two network interfaces in the same subnet use the same gateway, then you might encounter issues. To make sure that the kernel responds to ARP requests on the correct interface, run the following command to set ARP parameters:

sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2

To make your configuration persistent, run the following command:

cat << 'EOF' | sudo tee /etc/sysctl.d/90-multi-eni-arp.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
EOF
sudo sysctl --system

Confirm that routing works as expected

Run the following command to verify that the interface is up and has the expected IP addresses:

ip -br addr show

Run the following command to check the policy rules:

ip rule show

Run the following command to check the network interface routing table:

ip route show table 10001

Run the following command to check the network interface that packets use:

ip route get 8.8.8.8 from 10.0.2.20

Example output:

#   8.8.8.8 from 10.0.2.20 via 10.0.2.1 dev ens6 table 10001

Run the following command to confirm that default (unsourced) traffic uses the primary network interface:

ip route get 8.8.8.8

Example output:

#   8.8.8.8 via 10.0.1.1 dev ens5 src 10.0.1.10

Related information

How do I migrate my Amazon EC2 instance that runs AL2 to AL2023?