What is the default routing behavior with AWS Direct Connect Private and Transit Virtual Interfaces when identical routes are received from multiple Direct Connect locations?

Architecture setup

Figure 1 shows the base architecture we’ll be using in this article.

Figure 1: Base architecture

• This setup has four on-premises data centers connected to AWS at four different Direct Connect locations as follows:
  • DC1 (Seattle) → Seattle DX Location (Homed to us-west-2)
  • DC2 (Ashburn) → Ashburn DX Location (Homed to us-east-1)
  • DC3 (Dallas) → Dallas DX Location (Homed to us-east-1)
  • DC4 (London) → London DX Location (Homed to eu-west-2)

Refer to the Significance of the Associated Region when working with AWS Direct Connect article to understand the significance of an associated Region of a Direct Connect location when building hybrid infrastructure with AWS Direct Connect. To identify the home Region association for a DX location, refer to the AWS Direct Connect Locations page.

• The data centers are interconnected to each other through an MPLS/private network.
• In AWS, there are Amazon Virtual Private Clouds (VPCs) deployed in three Regions: us-west-2 (Oregon), us-east-1 (N. Virginia) and eu-west-2 (London). The VPCs are associated with respective Virtual Private Gateways (VGWs) or Transit Gateways (TGWs) in each Region. The VGWs/TGWs are in turn associated with a single DXGW. Note that a DXGW can either have VGW associations or TGW associations but not both at the same time. Also, there are quotas associated with the number of VGW or TGW associations you can have with a single DXGW. Consult the AWS Direct Connect quotas documentation to understand the limits and to choose the right gateway type that fits your needs.
• VIFs are configured on each DX connection with exterior Border Gateway Protocol (eBGP) sessions created between the customer/partner router and the DXGW. These VIFs can be private VIFs (if the DXGW is associated with VGWs) or transit VIFs (if the DXGW is associated with TGWs).
• A 10.0.0.0/8 aggregate route is being advertised over each VIF eBGP session toward the DXGW from the on-premises data center routers. The routes received on each VIF are identical with the same BGP AS PATH lengths, Multi-Exit Discriminator (MED) values, and other attributes. No Local Preference BGP communities have been applied to any of the advertised prefixes.

Routing behavior

Let's examine the default routing behavior from AWS VPCs to the on-premises 10.0.0.0/8 network in the following scenarios.

1. Scenario 1: When all four DX connections/locations are operational
2. Scenario 2: When Seattle DX connection/location is unavailable
3. Scenario 3: When Ashburn and Dallas DX connections/locations are unavailable
4. Scenario 4: When London DX connection/location is unavailable

Scenario 1: When all four DX connections/locations are operational

Figure 2: Routing behavior from AWS to on-premises data centers when all four connections/locations are operational

When all four DX connections/locations are operational,

• Traffic from us-west-2 (Oregon) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in Seattle location as shown in Figure 2 (Blue dotted arrow). This routing occurs because Seattle DX location is homed to us-west-2 (Oregon) Region and AWS will prioritize the home Region for outbound traffic.
• Traffic from us-east-1 (N. Virginia) VPCs to the on-premises 10.0.0.0/8 network will use Equal-Cost Multi-Path (ECMP) routing across the DX connections and VIFs in both Ashburn and Dallas locations (DC2 and DC3), as shown in Figure 2 (Green dotted arrows). This occurs because Ashburn and Dallas DX locations are homed to us-east-1 (N. Virginia) Region, and AWS sets equal priority to outbound traffic for routes received from different locations homed to the same Region.
• Traffic from eu-west-2 (London) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in the London location, as shown in Figure 2 (Red dotted arrow). This routing occurs because London DX location is homed to eu-west-2 (London) Region, and AWS will prioritize the home Region for outbound traffic.

Scenario 2: When Seattle DX connection/location is unavailable

Figure 3: Routing behavior from AWS to on-premises data centers when Seattle DX connection/location is unavailable

When Seattle DX connection/location becomes unavailable, AWS stops receiving the 10.0.0.0/8 network through its VIF from DC1. In this scenario,

• Traffic from us-west-2 (Oregon) VPCs to the on-premises 10.0.0.0/8 network will now use ECMP routing across the remaining three DX connections and VIFs, as shown in Figure 3 (Blue dotted arrows). This occurs because AWS sets equal priority to outbound traffic for routes received from locations not homed to the same Region as the VGW/TGW.
• As in scenario 1, traffic from us-east-1 (N. Virginia) VPCs to the on-premises 10.0.0.0/8 network will use ECMP across the DX connections and VIFs in both Ashburn and Dallas locations as shown in Figure 3 (Green dotted arrows).
• Also, like scenario 1, traffic from eu-west-2 (London) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in the London location (DC4) as shown in Figure 3 (Red dotted arrow).

Scenario 3: When Ashburn and Dallas DX connections/locations are unavailable

Figure 4: Routing behavior from AWS to on-premises data centers when Ashburn and Dallas DX connections/locations are unavailable

When both Ashburn and Dallas DX connections/locations become unavailable, AWS stops receiving the 10.0.0.0/8 network through their VIFs (DC2 and DC3). In this scenario,

• Traffic from us-west-2 (Oregon) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in Seattle location because DC1 is homed to us-west-2 (Oregon) Region, as shown in Figure 4 (Blue dotted arrow).
• Traffic from us-east-1 (N. Virginia) VPCs to the on-premises 10.0.0.0/8 network will now use ECMP routing across the DX connections and VIFs in both Seattle and London locations as shown in Figure 3 (Green dotted arrows). This occurs because AWS sets equal priority to outbound traffic for routes received from locations not homed to the same Region as the VGW/TGW.
• Traffic from eu-west-2 (London) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in London location as shown in Figure 3 (Red dotted arrow). This occurs because DC4 is homed to eu-west-2 (London) Region.

Scenario 4: When London DX connection/location is unavailable

Figure 5: Routing behavior from AWS to on-premises data centers when London DX connection/location is unavailable

When London DX connection/location becomes unavailable, AWS stops receiving the 10.0.0.0/8 network through its VIF (DC4). In this scenario,

• Traffic from us-west-2 (Oregon) VPCs to the on-premises 10.0.0.0/8 network will use the DX connection and VIF in Seattle location as shown in Figure 5 (Blue dotted arrow). This occurs because DC1 is homed to us-west-2 (Oregon) Region.
• Traffic from us-east-1 (N. Virginia) VPCs to the on-premises 10.0.0.0/8 network will use ECMP routing across the DX connections and VIFs in both Ashburn and Dallas locations, as shown in Figure 5 (Green dotted arrows). This occurs because DC2 and DC3 are homed to us-east-1 (N. Virginia) Region.
• Traffic from eu-west-2 (London) VPCs to the on-premises 10.0.0.0/8 network will now use ECMP routing across the remaining three DX connections and VIFs as shown in Figure 5 (Red dotted arrows). This occurs because AWS sets equal priority to outbound traffic for routes received from locations not homed to the same Region as the VGW/TGW.

Conclusion

In this article, you learned about the default routing behavior for traffic from AWS to on-premises networks when identical prefixes are received from multiple sources. You can override this default routing behavior by advertising more specific prefixes, unequal AS-PATH lengths, and/or implementing local preference BGP communities. For more information, refer to the following documentations.

• AWS Direct Connect routing policies and BGP communities
• Active/Active and Active/Passive Configurations in AWS Direct Connect
• Creating active/passive BGP connections over AWS Direct Connect
• How do I set an Active/Passive Direct Connect connection to AWS?

About the author

Mandar is a Senior Networking Solutions Architect at AWS. He is passionate about networking technologies and loves to innovate and help solve complex customer problems. He holds a master’s degree in Telecommunications from University of Colorado Boulder. Mandar lives in Seattle and loves travel and outdoor activities.