How do I calculate the total network usage for my Amazon EC2 instances and determine which instance used the most bandwidth within a date range?

Short description

To determine which EC2 instance causes high network usage, use the Amazon CloudWatch NetworkIn and NetworkOut metrics. You can combine the data points from the NetworkIn and NetworkOut metrics to calculate the network usage for your EC2 instance.

Note: The CloudWatch NetworkOut metric is a different metric from the DataTransfer-Out-Bytes metric that's used in AWS Cost Explorer reports. The CloudWatch NetworkOut metric represents the amount of outbound network traffic from an instance.

Prerequisites:

• Set up an AWS Command Line Interface (AWS CLI) environment. For more information, see What is the AWS Command Line Interface?
• Turn on the cloudwatch:GetMetricStatistics permission for the AWS Identity and Access Management (IAM) user or role that you use to run the AWS CLI. For more information, see Actions, resources, and condition keys for Amazon CloudWatch.
• Have access to the Bash shell to run the script.

Note: The following resolution works only for instances that you can see in your Amazon EC2 dashboard. The following Bash shell script doesn't work for terminated instances. For terminated instances, add the instance IDs to the script in a space-separated format.

Example space-separated format:

ADD_INSTANCES="i-aaaa i-yyyy i-zzzz"

Resolution

Note: If you receive errors when you run AWS CLI commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To identify the network usage of all EC2 instances in an AWS Region, complete the following steps:

1. Create a new script file, and then copy the following script into the file:

   #!/bin/bash
   if [ $# -ne 4 ]; then
       echo "Usage: $0 <REGION> <NetworkIn|NetworkOut> <START_TIMESTAMP> <END_TIMESTAMP>"
       echo -e "\tNote: Do not change the order of parameters."
       echo -e "\n\tExample: $0 ap-south-1 NetworkOut 2020-06-01T00:00:00.000Z 2020-06-30T23:59:59.000Z"
       exit 1
   fi
   
   REGION="$1"
   METRIC="$2"
   START_TIME="$3"
   END_TIME="$4"
   
   # Add specific instances if needed
   ADD_INSTANCES=""
   
   # Combine ADD_INSTANCES with running instances and remove duplicates
   INSTANCES=$(echo "${ADD_INSTANCES} $(aws ec2 describe-instances --region ${REGION} --query Reservations[*].Instances[*].InstanceId --output text)" || { echo "Failed to run aws ec2 describe-instances commandline, exiting..."; exit 1; } | tr ' ' '\n' | sort -u | tr '\n' ' ')
   
   [ "${INSTANCES}x" == "x" ] && { echo "There are no instances found from the given region ${REGION}, exiting..."; exit 1; }
   
   # Array to store results
   declare -A results
   
   # Collect network usage data
   for _instance_id in ${INSTANCES}; do
       unset _value
       _value="$(aws cloudwatch get-metric-statistics --metric-name ${METRIC} \
           --start-time ${START_TIME} \
           --end-time ${END_TIME} \
           --period 86400 \
           --namespace AWS/EC2 \
           --statistics Sum \
           --dimensions Name=InstanceId,Value=${_instance_id} \
           --region ${REGION} \
           --output text)"
       
       if [ "${_value}x" == "x" ]; then
           echo "Something went wrong while calculating the network usage of ${_instance_id}"
           continue
       fi
       
       # Calculate usage in GiB and store in results array
       usage=$(echo "${_value}" | awk '{ sum += $2 } END {printf ("%.6f\n", sum/1024/1024/1024)}')
       results[$_instance_id]=$usage
   done
   # Print results sorted by usage (highest to lowest)
   echo -e "\nNetwork usage by instance (sorted by usage):"
   echo "----------------------------------------"
   for instance in "${!results[@]}"; do
       echo "${instance}: ${results[$instance]} GiB"
   done | sort -rn -k2
   echo -e "\nNote: If you think the values are inaccurate, please verify the input and modify if needed."

2. Save the script with a new name. For example, network_usage.sh.

3. To turn on the permission for the script to run, run the following command:

   sudo chmod a+x network_usage.sh

   Note: Replace network_usage.sh with your script name.

4. Run the script with the following syntax:

   bash network_usage.sh ap-south-1 NetworkOut 2020-06-01T00:00:00.000Z 2020-06-30T23:59:59.000Z

   Note: Replace ap-south-1 with the Region where your instances are located. Specify either NetworkIn or NetworkOut to calculate the traffic flow direction. Then, replace 2020-06-01T00:00:00.000Z and 2020-06-30T23:59:59.000Z with the start and end timestamps in the range that you want to calculate the network usage for.
   Example output:

   Network usage by instance (sorted by usage):
   ----------------------------------------
   i-0ef3921993739d772: 219.884000 GiB
   i-06c4d1e1cb4e9139d: 0.012911 GiB
   i-05ca8ccc8aa7caf76: 0.009235 GiB
   i-0b56df07325d9c0fa: 0.008475 GiB
   i-025a820f5ee9f1490: 0.000099 GiB
   i-09ad0a93b60a233b1: 0.000000 GiB
   i-0ba35b570c7003ffe: 0.000000 GiB
   i-012cc851406584b5c: 0.000000 GiB
   i-01f193931c4101cf3: 0.000000 GiB
   i-06b165356b5769581: 0.000000 GiB
   i-070b6508dd4ab5040: 0.000000 GiB
   i-0f29e3ba7f156dfee: 0.000000 GiB

Related information

CloudWatch metrics that are available for your instances

GNU Bash on the GNU website