How do I calculate how many bytes are transferred over my VPC peering connection within custom VPC flow logs?

Resolution

Use Amazon CloudWatch Logs Insights and Amazon Athena to query Amazon VPC flow logs.

Prerequisites

Create an active Amazon VPC peering connection.

Activate Amazon VPC Flow Logs on at least the source VPC. You can publish VPC flow logs to CloudWatch Logs or an Amazon Simple Storage Service (Amazon S3) bucket.

This example custom VPC Flow Logs format uses the ${traffic-path} and ${bytes} fields:

${interface-id} ${vpc-id} ${srcaddr} ${dstaddr} ${pkt-srcaddr} ${pkt-dstaddr} ${traffic-path} ${bytes} ${log-status}

To determine how many bytes are transferred over a peering connection, specify and query the ${traffic-path} and ${bytes} fields. The traffic-path identifies the path that egress traffic takes to the destination. 

Note: The traffic path values for Amazon VPC peering connections are 4 for intra-AWS Region and 5 for inter-AWS Region.

For more information, see Available fields.

Use CloudWatch Insights to query flow logs

Follow these steps to use CloudWatch Logs Insights to query flow logs:

1. Open the CloudWatch console.

2. In the navigation pane, choose Logs Insights.

3. On the Logs Insights dashboard, choose the log group.

4. Run this query to calculate the total bytes that are transferred over all intra-Region and inter-Region VPC peering connections within the VPC log:

   fields @timestamp, bytes
   |filter trafficPath in ['4','5']
   | stats sum(bytes) as
   TotalBytesTransferred

5. Run this query to calculate the total number of bytes that are transferred between the source and destination over all intra-Region and inter-Region VPC peering connections. The results are listed in descending order based on the number of bytes that are transferred:

   fields @timestamp, bytes, srcAddr, dstAddr
   | sort srcAddr, dstAddr
   | filter trafficPath in ['4', '5']
   | stats sum(bytes) as TotalBytesTransferred by srcAddr, dstAddr
   | sort -TotalBytesTransferred
   

6. Run this query to search by the source and destination IP address:

   fields @timestamp, bytes, srcAddr, dstAddr
   | sort @timestamp desc
   | filter trafficPath in ['4', '5']
   | filter srcAddr = "x.x.x.x" and dstAddr = "x.x.x.x"
   | stats sum(bytes) as TotalBytesTransferred by srcAddr, dstAddr
   | sort -TotalBytesTransferred

For custom queries, see Tutorial: Run a query with an aggregation function. For information on query syntax, see CloudWatch Logs Insights query syntax.

For more information on how to use CloudWatch Logs Insights queries, see How can I use CloudWatch Logs Insights queries with my VPC flow log?

Use Amazon Athena to query VPC flow logs

Follow these steps to use Amazon Athena to query VPC flow logs:

Note: By default, VPC flow logs are activated in text format with an Amazon S3 bucket as the destination.

1. Open the Amazon Athena console, and then launch the query editor.

2. Create an Athena table for Amazon VPC flow logs.

3. Use a DDL statement to create a table with columns that match the data in your VPC flow logs. Make sure that the line numbers align with the column numbers in your Amazon S3 file's output. Use the Amazon S3 bucket URI for the location source.

4. For Amazon S3 default text formatted queries, use this DDL statement:

   CREATE EXTERNAL TABLE IF NOT EXISTS `vpc_flow_logs` (
     interface_id string,
     vpc_id string,
     srcaddr string,
     dstaddr string,
     pkt_srcaddr string, 
     pkt_dstaddr string, 
     traffic_path int, 
     bytes bigint, 
     log_status string
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ' '
   LOCATION 's3://EXAMPLE_BUCKET/REGION/'
   TBLPROPERTIES ("skip.header.line.count"="1");

   Note: Each table column is added in the order of entry. The first column is interface_id string because the last line omits the header from the count.

5. Run this query to determine the total bytes that are transferred over an intra-Region or inter-Region peering connection:

   SELECT sum(bytes) as total_bytes_transferred
   FROM vpc_flow_logs
   WHERE traffic_path IN (4, 5);

6. Run this query to calculate the total bytes that are transferred over an intra-Region or inter-Region peering connection:

   SELECT
       srcaddr,
       dstaddr,
       SUM(bytes) AS total_bytes_transferred
   FROM
       vpc_flow_logs
   WHERE traffic_path IN (4,5)
   GROUP BY
       srcaddr, dstaddr
   ORDER BY
       total_bytes_transferred DESC;

Note: The data is categorized by source and destination IP address and listed in descending order based on the number of bytes that are transferred.

For more information, see Creating and querying tables for custom VPC flow logs.

Related information

Logging IP address traffic using VPC flow logs

Publish flow logs to CloudWatch logs

Publish flow logs to Amazon S3