How do I use EC2Rescue for Linux to troubleshoot performance bottlenecks in my Amazon EC2 Linux instances?

Short description

To determine where performance bottlenecks occur, use the bcc framework (eBPF) in EC2Rescue for Linux to check for CPU, storage, or network issues. For instances that run Ubuntu, use the paired EC2Rescue modules that have names that begin with bpfcc.

Note: The following resolution doesn't apply to instances that run Debian.

Resolution

Install the bcc package

Complete the following steps:

1. Use SSH to connect to your instance.
2. To install the bcc package, run the following command based on your distribution.
   AL2023:

   sudo dnf install bcc-tools

   AL2:

   sudo amazon-linux-extras enable BCC 

   sudo yum install kernel-devel-$(uname -r) bcc

   Note: AL2 reached its end of life on June 30, 2026. If you run AL2, then migrate to AL2023 to maintain a supported version.
   RHEL 8 or 9:

   sudo dnf install bcc-tools libbpf 

   Ubuntu:

   sudo apt install bpfcc-tools linux-headers-$(uname -r) 
   

   Note: For prerequisites such as matching kernel headers, see the BCC installation guide for your distribution.
3. For Amazon Linux and RHEL instances, run the following command to confirm that the BCC executable that EC2Rescue runs is available:

   command -v runqlat

   Note: For Ubuntu instances, add -bpfcc to the end of the command. For Ubuntu instances, the executable names must have the -bpfcc suffix.
4. If the command returns no output, then the BCC executable is missing or in the wrong PATH. To resolve this issue, reinstall the distribution package. If the package is installed outside of the PATH, then use the package manager to locate it. Then, add the directory to the current session PATH. Rerun the preceding command to confirm that the BCC executable is available.
5. Install the EC2Rescue for Linux tool.

Use bcc-based modules with EC2Rescue for Linux

Run commands

Each of the following modules runs for the specified period and collects output a specified number of times. After you run a module on your instance, you can find the output of these modules under the /var/tmp/ec2rl directory.

Example run command for Amazon Linux or RHEL instances:

sudo ./ec2rl run --only-modules=bcctcptop --period=5 --times=2

Note: For Ubuntu instances, replace bcctcptop with bpfcctcptop.

EC2Rescue stores the module output in /var/tmp/ec2rl.

If you must upload the results of the run command for AWS Support, then confirm that the collected files don't contain confidential information. Then, run the following command to upload the results:

sudo ./ec2rl upload --upload-directory=results-directory --support-url='URL-provided-by-AWS-Support'

Note: Replace results-directory with the results directory and URL-provided-by-AWS-Support with the upload URL that AWS Support provided.

CPU performance tools

The bccsoftirqs.yaml module runs the softirqs tool that traces soft interrupts (IRQs), and then stores timing statistics in the kernel. You can use --period to provide an interval, and --times for a count. The tool automatically shows the timestamps for each time that it runs. For Ubuntu, the module's named bpfccsoftirqs. For more information, see aws-ec2rescue-linux/mod.d/bccsoftirqs.yaml on the GitHub website.

The bccrunqlat.yaml module shows the time that tasks are waiting to run on CPU as a histogram. For Ubuntu, the module's named bpfccrunqlat. For more information, see aws-ec2rescue-linux/mod.d/bccrunqlat.yaml on the GitHub website.

Block I/O performance tools

The bccbiolatency.yaml module traces block device I/O and records the distribution of I/O latency for each disk device as a histogram. For Ubuntu, the module's named bpfccbiolatency. For more information, see aws-ec2rescue-linux/mod.d/bccbiolatency.yaml on the GitHub website.

The bccext4slower.yaml module uses the ext4slower tool to collect output. The tool traces ext4 reads, writes, opens, and fsyncs that are slower than a threshold of 10 milliseconds. For Ubuntu, the module's named bpfccext4slower. For more information, see aws-ec2rescue-linux/mod.d/bccext4slower.yaml on the GitHub website.

Note: For XFS file systems, use the bccxfsslower module instead of bccext4slower.yaml. For Ubuntu, the module's named bpfccxfsslower. For more information, see aws-ec2rescue-linux/mod.d/bccxfsslower.yaml on the GitHub website.

The bccfileslower.yaml module uses fileslower to collect output. The tool traces file-based synchronous reads and writes that are slower than a default threshold of 10 milliseconds. For Ubuntu, the module's named bpfccfileslower. For more information, see fileslower_example and aws-ec2rescue-linux/mod.d/bccfileslower.yaml on the GitHub website.

Network performance tools

The bcctcpconnlat.yaml module traces the kernel function that performs active TCP connections, such as through a connect() syscall. The results show the latency for the connection. Latency is locally measured as the time from SYN that's sent to the response packet for a specified period. TCP connection latency shows the time that it takes to establish a connection. For Ubuntu, the module's named bpfcctcpconnlat. For more information, see aws-ec2rescue-linux/mod.d/bcctcpconnlat.yaml on the GitHub website.

The bcctcptop.yaml module shows TCP connection throughput for each host and port for the specified period and times and doesn't clear the screen.For Ubuntu, the module's named bpfcctcptop. For more information, see aws-ec2rescue-linux/mod.d/bcctcptop.yaml on the GitHub website.

The bcctcplife.yaml module summarizes TCP sessions that open and close when the session is tracing. For Ubuntu, the module's named bpfcctcplife. For more information, see aws-ec2rescue-linux/mod.d/bcctcplife.yaml on the GitHub website.

Related information

How do I troubleshoot high CPU usage on an Amazon EC2 Windows instance?

How do I diagnose high CPU usage on my Amazon EC2 Windows instance when my CPU doesn't throttle?

How do I troubleshoot high CPU utilization on an Amazon EC2 Linux instance?