How do I configure Traffic Mirroring to troubleshoot network connectivity for my Application Load Balancer?

Short description

Traffic Mirroring in your Amazon Virtual Private Cloud (Amazon VPC) captures network traffic from your Application Load Balancer's elastic network interfaces (ENIs). Traffic Mirroring sends the traffic to a target Amazon Elastic Compute Cloud (Amazon EC2) instance for analysis.

Resolution

Prerequisites: To use Traffic Mirroring, confirm that your Application Load Balancer and target EC2 instance are in the same Amazon VPC. Then, assign permissions to create Amazon VPC Traffic Mirroring resources. For more information about traffic mirroring prerequisites, see Get started using Traffic Mirroring to monitor network traffic.

Launch and configure the target Amazon EC2 instance

Launch an Amazon Linux 2 EC2 instance in the same Amazon VPC as your Application Load Balancer. To configure the instance with the following specifications, complete the following steps:

1. Open the Amazon EC2 console.
2. Choose Launch Instance.
3. Choose Amazon Linux 2 as the Amazon Machine Image (AMI).
4. Choose an instance type that has the appropriate network performance for your traffic volume.
5. Configure the instance to launch in the same Amazon VPC as your Application Load Balancer.
6. Allocate at least 200 GB of storage to accommodate packet capture files.
7. Configure the security group to allow the following traffic.
   For Inbound, use UDP port 4789 over Virtual Extensible LAN (VXLAN) from your Amazon VPC CIDR range.
   For Outbound, use SSH on port 22 from your IP address for management access.
8. Launch the instance, and then verify that you can connect to the instance using SSH.

Create a traffic mirror target and filter

To create a target and filter for your traffic mirror, complete the following steps:

1. Create a traffic mirror target that points to the ENI of your target Amazon EC2 instance.
2. Create a traffic mirror filter that captures all traffic.
3. For your traffic mirror filter, under Inbound rules, choose Add rule.
   For Rule number, enter 100.
   For Rule action, choose accept.
   For Protocol, choose All protocols.
   For Source CIDR block, enter 0.0.0.0/0.
   For Destination CIDR block, enter 0.0.0.0/0.
   Leave Source port range and Destination port range blank.
4. Under Outbound rules, configure a rule with the same settings as the inbound rule.

Note: The traffic mirror filter configuration captures all inbound and outbound traffic. For production environments, you must create more specific filters to capture only the traffic that you want to analyze.

Create traffic mirror sessions for each Application Load Balancer ENI

Application Load Balancers have one ENI per subnet. You need to create a separate traffic mirror session for each Application Load Balancer ENI.

To identify the ENIs associated with your Application Load Balancer, complete the following steps:

1. Open the Amazon EC2 console.
2. Choose Load Balancers in the navigation pane.
3. Select your Application Load Balancer.
4. In the Description tab, note the Availability Zones section to identify the subnets.
5. Choose Network Interfaces in the navigation pane.
6. Filter by the Application Load Balancer name or subnet IDs to find the associated ENIs.

To create a traffic mirror session for the first Application Load Balancer ENI, complete the following steps:

1. Open the Amazon VPC console.
2. Choose Traffic Mirroring, and then choose Mirror Sessions.
3. Choose Create traffic mirror session.
   For Name tag, enter a descriptive name. For example, enter Application Load Balancer-ENI-1-Mirror.
   For Mirror source, select the first Application Load Balancer ENI.
   For Mirror target, select the traffic mirror target that you created in step 2 of the Create a traffic mirror target and filter section.
   For Filter, select the traffic mirror filter that you created in step 3 of the Create a traffic mirror target and filter section.
   For Session number, enter a unique number. For example, enter 1.
   For VNI (VXLAN Network Identifier), enter a unique number. For example, enter 100.
   Note: Record the VNI value for later use.
4. Choose Create.

Then, repeat the preceding steps for each additional ENI for your Application Load Balancer. Make sure that you use unique session numbers and VNI values for each session. For example, for your second ENI, for Session number, enter 2, and then for VNI, enter 200. Record the VNI values because you need them in the next step.

For more information, see Create, modify, or delete a traffic mirror session.

Configure VXLAN interfaces on the target instance

Connect to your target instance with SSH, and then create VXLAN interfaces to decapsulate the mirrored traffic. Complete the following steps:

1. To identify your target instance's ENI details, run the following Linux command:

   ip addr show 

   Note: Note the device name. For example, note eth0. Also, note the IP address of the primary network interface.

2. To create a VXLAN interface for each Traffic Mirror session, run the following Linux command:

   ip link add vxlan1 type vxlan id VNI_FOR_ENI_1 dev DEVICE dstport 4789 local IP 
   ip link set vxlan1 up 
   
   ip link add vxlan2 type vxlan id VNI_FOR_ENI_2 dev DEVICE dstport 4789 local IP
   ip link set vxlan2 up 
   
   ip link add vxlan3 type vxlan id VNI_FOR_ENI_3 dev DEVICE dstport 4789 local IP
   ip link set vxlan3 up 

   Note: Replace VNI_FOR_ENI_# with the VNI value that you specified when you created each Traffic Mirroring session. Replace DEVICE with the ENI device name, and replace IP with the IP address of the target instance's ENI.

3. To verify that the VXLAN interface setup succeeded, run the following Linux command:

   ip link show | grep vxlan 

Capture and analyze traffic

Capture traffic from each VXLAN interface. Make sure that you open separate SSH sessions to the target instance to run these commands concurrently.

For example, make sure that you run the following commands to capture traffic for a series of 3 SSH sessions, each with its own Application Load Balancer ENI:

• For session 1, run the following command:

  tcpdump -i vxlan1 -W 50 -C 200 -w /tmp/capture_ENI1.pcap 

• For session 2, run the following command:

  tcpdump -i vxlan2 -W 50 -C 200 -w /tmp/capture_ENI2.pcap 

• For session 3, run the following command:

  tcpdump -i vxlan3 -W 50 -C 200 -w /tmp/capture_ENI3.pcap 

Note: The -W and -C flags in these examples create 50 rotating capture files with a file size of 200 MB and a maximum total size of approximately 10 GB. When you reach the limit, tcpdump overwrites the oldest files. Adjust these values based on your available disk space and expected traffic volume.

Analyze the captured traffic

After you capture the traffic, download the Packet Capture (PCAP) files from the target instance to your local machine for analysis:

1. Use Secure Copy Protocol (SCP) or another file transfer method to download the capture files:

   scp -i your-key.pem ec2-user@target-instance-ip:/tmp/capture_ENI*.pcap /local/path/  

   Note: Replace your-key.pem with the name of your Privacy-Enhanced Mail (PEM) file and target-instance-ip with the IP of your target instance.
2. Open the PCAP files with a network protocol analyzer, such as Wireshark, to inspect the traffic. For more information, see Open Capture Files on the Wireshark website.
3. Look for patterns that identify your Application Load Balancer issues, such as malformed packets or dissector bugs. For more information, see Expert Information on the Wireshark website.

Clean up resources

To delete the Traffic Mirroring resources to avoid ongoing charges, complete the following steps:

1. Open the Amazon VPC console.
2. Choose Traffic Mirroring, and then choose Mirror Sessions.
3. Select each traffic mirror session that you created, then choose Actions, and then choose Delete.
4. Choose Mirror Filters, and then select your filter.
5. Choose Actions, Delete.
6. Choose Mirror Targets, and then select your target.
7. Choose Actions, Delete.
8. (Optional) Terminate the target instance if you no longer need it.

Related information

What is Traffic Mirroring?

What is an Application Load Balancer?

Troubleshoot your Application Load Balancers