Enabling VMware Cloud on AWS workloads to access native AWS Services over Layer 2 Extended (L2E) Segments

11 minute read
Content level: Advanced
3

Using Route Aggregation and HCX MON to enable SDDC L2E segments to access the connected Amazon VPC

Introduction

This post will guide you through how to enable VMware Cloud on AWS workloads on layer 2 extended (L2E) segments to access AWS native services in the connected Amazon VPC, by using Route Aggregation and HCX Mobility Optimized Networking (MON).

VMware Cloud on AWS offers one of the fastest ways to migrate on-premises vSphere workloads to the cloud, while minimizing migration complexity and risks. Using VMware HCX, customers are able to extend Layer 2 networks from their on-premises data centers into VMware Cloud on AWS, allowing virtual machines (VMs) to retain their IP addresses during cloud migrations. In addition, VMware Cloud on AWS customers can seamlessly integrate their SDDCs with native AWS services. During an SDDC onboarding process, customers are able to establish high-bandwidth and low-latency connectivity to a connected VPC, via a cross-account Elastic Network Interface (xENI).

However, by default it is not possible for workloads running on L2E segments to access native AWS services in the connected VPC, even though they are already migrated to VMware Cloud on AWS. This is due to there is no return routes of those L2E prefixes in the connected VPC route table.

Enter image description here

One possible workaround is to configure a second Virtual Network Interface Card (vNIC) on individual VMs over a natively routed SDDC segment. However, this option does introduce additional configuration overhead and management complexity.

A recently introduced new option is to leverage Route Aggregation and HCX Mobility Optimized Networking (MON). MON improves network performance and reduces latency for virtual machines that have been migrated to the cloud on an L2E network. More details on HCX MON use cases and design patterns are documented here. You can use HCX MON to directly forward L2E segments traffic to the connected VPC over the xENI, instead of tromboning back through on-premises gateway.

In addition, the ability to configure route aggregation was introduced in VMware Cloud on AWS version 1.18. Customers can now leverage route aggregation and AWS-managed prefix list to advertise L2E prefixes from their SDDCs to the connected VPC route table, allowing a symmetric return path via the xENI.

Enter image description here

Through this post, I will provide an example with a detailed walkthrough for enabling SDDC workloads on L2E segments to access Amazon FSx fileshare service in the connected VPC, by configuring route aggregation and HCX MON.  

Pre-requisite

  • 1x VMware Cloud on AWS SDDC (ver. 1.18+) with at least one routed segment
  • 1x on-premises vSphere Lab environment (min 6.5+, see HCX support for legacy vSphere environment)
  • AWS Direct Connect or IPsec VPN L3 connectivity between on-premises and SDDC (refer to HCX underlay minimum requirements)
  • HCX deployed with Service-Mesh configured between on-premises and SDDC
  • 1x L2E segment from on-premises to SDDC
  • 1x Connected VPC with the following native services deployed
    • 1x Amazon FSx file system
    • 1x AWS Managed Microsoft AD (optional, for FSx authentication/authorization)
  • 2x Windows 2019 test VMs
    • 1x VM deployed in the SDDC natively routed segment
    • 1x VM migrated to the SDDC over the HCX L2E segment

To learn how to deploy and configure HCX with L2E, please refer to the previous re:Post article for more details.  

Setup

For this example, I’ll walk through the configurations covering the following sections:

  1. Review demo environment network setup
  2. Enable AWS Managed Prefix List for the Connected Amazon VPC
  3. Configure SDDC Route Aggregation
  4. Configure HCX MON  

Part1: Review demo environment network setup

To begin, let's review the current demo environment network setup.

Here I have deployed a SDDC with two routed segments (10.200.0.0/24, 10.200.1.0/24). I have also prepared an on-premises vSphere lab (not shown) with HCX deployed and Service Mesh configured between the two environments. In addition, an on-premises layer 2 segment (10.10.11.0/24) is extended via HCX into the SDDC. At this stage, I have not enabled HCX MON yet so it is showing "Disconnected" to the SDDC local Compute Gateway (CGW).

Enter image description here

At the connected VPC default route table, we can see the two SDDC routed segments and one extra SDDC management route (10.252.0.0/20). All prefixes are automatically programmed towards the SDDC xENI as expected. However, notice the L2E prefix is missing since we have not configured SDDC route aggregation. This prevents VM workloads on the L2E network from accessing resources in the connected VPC, as there is no valid return route.

Enter image description here

For testing purposes, I have also prepared an Amazon FSx file system in the connected VPC, as well as two Windows Server 2019 test VMs:

  • 1x test VM (10.200.0.100) deployed in the SDDC on a routed segment
  • 1x test VM (10.10.11.100) migrated to the SDDC over the HCX L2E segment

As expected, the first VM (10.200.0.100) was able to access FSx via the xENI but the second VM (10.10.11.100) could not reach the fileshare service.

Enter image description here  

Part2: Enable AWS Managed Prefix List for the Connected Amazon VPC

One of the pre-requisites to configure route aggregation towards the connected VPC is to enable AWS Managed Prefix List for the Connected VPC. Because it is a managed object, the prefix list gets updated automatically whenever new segments or route aggregations are configured. This allows customers to configure and advertise selected L2E prefixes to the connected VPC for accessing native services over the xENI.

To enable Managed Prefix List for the connected VPC, navigate to the Networking & Security tab within the SDDC console, and go to Systems > Connected VPC then toggle the switch AWS Managed Prefix List Mode.

Enter image description here

Next, go to the AWS account of the connected VPC and navigate to Resource Access Manager (RAM). You should see a managed prefix list resource being shared from the (VMware managed) shadow VPC account. Go ahead and accept the resource share.

Enter image description here

After a few minutes you should see the managed prefix list becomes available within the SDDC console. It is now applied to the connected VPC default route table, simplifying route table management by replacing the individual SDDC routes.

Enter image description here

The prefix list contains all three SDDC routes as identified previously.

Enter image description here  

Part3. Configure SDDC Route Aggregation

The SDDC route aggregation is only configurable via the NSX Manager UI, which is accessible within the SDDC console via OPEN NSX MANAGER.

Enter image description here

Within the NSX Manager UI, navigate to Networking > Settings/Global Configuration > Route Aggregation.

We will first configure an Aggregation Prefix List to include the following prefixes:

  • 10.10.0.0/20 (summarized prefix for all HCX L2E segments)
  • 10.200.0.0/20 (summarized prefix for all SDDC routed workload segments)

Note the SDDC management prefix is always explicitly advertised and cannot be summarized, so it is not included in the aggregation list.

Enter image description here

Next, we will apply the aggregation prefix list and advertise it to the connected VPC. This is achieved by selecting the SERVICES connectivity endpoint as shown below.

Enter image description here

To verify results, navigate to Networking > Cloud Services/Connected VPC > Advertised, you should see the aggregated prefixes for the native SDDC segments and HCX L2E segments, as well as the explicitly advertised SDDC management network.

Enter image description here

Back at the default route table for the connected VPC, you can see the managed prefix list is now updated with the aggregated prefixes as advertised from the SDDC side. This provides a symmetric return route for SDDC traffic from L2E segments via the xENI.

Enter image description here  

Part4. Configure HCX MON

To enable VM workloads to access native services in the connected VPC, the last step is to configure HCX MON. This will allow us to drop traffic from L2E segments to the local CGW and Tier-0 router, and then forward it to the connected VPC via the xENI.

Navigate to the HCX Manager UI, locate the L2E segment (10.10.11.0/24 in my case) under Services > Network Extension, and toggle the switch of Mobility Optimized Networking.

Enter image description here

With MON enabled, HCX automatically sets the Target Router Location to the CGW (T1) for VMs that have been bulk migrated or DR recovered to the SDDC. For VMs migrated using HCX vMotion or Replication Assisted vMotion (RAV), or VMs attached to the L2E segment prior to enabling MON, it is required to manually set the target router location.

Since my test VM (HCX-Test-Win01) was migrated prior to enabling MON, I will select the VM from the list and set the Router Location to the SDDC and click SUBMIT. Wait until the MON configuration is completed for the VM, and verify its target router is updated to the SDDC side (CGW).

Enter image description here

As per the HCX user guide, MON uses policy routes to decide when to send the traffic back via the original L2E path, or to drop it to the CGW/Tier-0 for local L3 forwarding. When the destination network for a traffic flow is not within the SDDC, the MON policy is evaluated:

  • If the destination IP is matched and configured as allow in the MON policy configuration, the packet is forwarded to the on-premises gateway via the HCX L2E path.
  • If the destination IP is not matched, or configured to deny in the MON policy, the packet is forwarded to the SDDC Tier-0 for local L3 forwarding.

You can refer to this technical whitepaper for recommendations and best practices for the HCX MON policy configurations.

Based on my setup, I will modify the MON Policy Routes table to explicitly deny the connected VPC CIDR (10.251.0.0/16), so traffic towards this range will be directly routed to the xENI via the local CGW and Tier-0. I have also added a single allow entry for network 0.0.0.0/0, so traffic towards all other networks (outside the SDDC) is forwarded back to the on-premises gateway via the L2E path to avoid potential asymmetric routing.

Enter image description here

We have now successfully enabled VM workloads from L2E segments to access resources in the connected VPC by using SDDC route aggregation and HCX MON. At the SDDC console, we can see the L2E segment is now showing Routed with MON enabled.

Enter image description here

The test VM running on the L2E is now able to discover and mount the FSx share deployed in the connected VPC, using the high-bandwidth and low-latency xENI.

Enter image description here  

Additional Considerations

  • Data transfer between the SDDC and connected VPC via the xENI will not incur any egress costs as long as the SDDC and destined native services reside in the same AWS Availability Zone (AZ).
  • MON requires VM tools to be installed on the guest OS.
  • MON is a HCX feature and does not provide traffic optimization for other L2E solutions such as Layer 2 VPN with NSX autonomous edge.
  • The solution discussed in this post only applies to traffic optimization between MON-enabled L2E segments and the connected VPC. MON does not provide traffic optimization between L2E segments and other SDDCs or native VPCs via the VMware Transit Connect. Refer to here for more details on MON limitations.
  • MON optimized traffic will be inspected by the CGW firewall, which is outside the scope of this post. You can read more details on how to configure CGW firewall rules at here.  

Conclusion

In this article you have learnt how to enable VMware Cloud on AWS workloads on L2E networks to access the connected Amazon VPC, by using SDDC route aggregation and HCX MON.  

Additional resources