Using lifecycle_scope with vTGW/TGW peering in VMware Cloud on AWS

Situation

My team runs a customer-facing Immersion Day, which is a hands-on event allowing customers to work in a live VMware Cloud on AWS SDDC, as well as work with native AWS services in a live AWS account. The VMC on AWS side of the lab contains 2 SDDCs connected via a VMware Transit Gateway (vTGW). The AWS native side of the lab contains student VPCs peered to a native Transit Gateway. The vTGW and native TGW are peered for seamless communication. The lab environment is provisioned with a combination of Terraform and Cloud Formation.

Task

My team has been using a Python script to create and destroy the vTGW-TGW peering connection. While this was OK, I wanted to bake the process into our Terraform code and eliminate the extra Python step.

Actions

I knew that I wanted my Terraform code to create and invoke a Lambda function that would perform the peering. I had not used Terraform with Lambda before, so I started searching the documentation. I found a way to create a Lambda function, and a way to invoke a Lambda function.

In the aws_lambda_invocation documentation, I found an option called lifecycle_scope. By default, a Lambda is only invoked on create or replacement. But the instruction lifecycle_scope = "CRUD" will cause the Lambda to be invoked on all lifecycle events. This new feature, published in May 2023, gave me exactly what I wanted: my function can now create the vTGW-TGW peering when we run terraform apply, and destroy the peering when we run terraform destroy

Listed below is how I defined the Lambda function and invocation in Terraform. You can see the Lambda input variables that I need to pass in order to make the VMC on AWS API call to initiate a vTGW-TGW peering. One item to highlight that really tripped me up is the line:

function_name = aws_lambda_function.vtgw_tgw_peering_lambda.function_name

.function_name is an actual property, not a placeholder in the documentation. I was reading the documentation thinking that .function_name was a placeholder and could not get it to work. I was trying to use:

# This is invalid
aws_lambda_function.vtgw_tgw_peering_lambda.vtgw_tgw_lambda

This does not work because you need to use an actual property name: .function_name.

resource "aws_lambda_function" "vtgw_tgw_peering_lambda" {
  filename      = "lambda/vtgw_tgw_lambda_payload.zip"
  function_name = "vtgw_tgw_lambda"
  role          = aws_iam_role.iam_role_for_lambda.arn
  handler       = "index.lambda_handler"
  runtime = "python3.11"
  timeout = 600
  memory_size = 512
  layers = [aws_lambda_layer_version.lambda_layer.arn]
}
resource aws_lambda_invocation "vtgw_lambda" {
  function_name = aws_lambda_function.vtgw_tgw_peering_lambda.function_name
  input = jsonencode(
    {
      "vmc_refresh_token" = var.vmc_refresh_token,
      "vmc_org_id" = data.terraform_remote_state.phase1.outputs.vmc_org_id,
      "sddc_a_region" =  data.terraform_remote_state.phase1.outputs.sddc_a_region,
      "sddc_a_id" = data.terraform_remote_state.phase1.outputs.sddc_a_id,
      "sddc_group_id" = data.terraform_remote_state.phase1.outputs.sddc_group_id,
      "aws_account_number" = data.terraform_remote_state.phase1.outputs.aws_account_number
    }
  )
  lifecycle_scope = "CRUD"
}

When you set lifecycle_scope="CRUD", Terraform injects a key named tf with a subkey named action to let you know which phase Terraform was in when invoking the Lambda.

This is a snippet of my Lambda handler showing how I read the action key and branched my code based on create/delete.

def lambda_handler(event, context):
        vmc_refresh_token = event['vmc_refresh_token']
        vmc_org_id = event['vmc_org_id']
        sddc_a_id = event['sddc_a_id']
        sddc_a_region = event['sddc_a_region']
        sddc_group_id = event['sddc_group_id']
        aws_account_number = event['aws_account_number']
        tf_action = event['tf']['action']

       print(f"tf_action = {tf_action}")
       match tf_action:
           case "create":
               print("Peering vTGW with TGW...")

           case "delete":
               print("Deleting vTGW/TGW peering...")

Result

The new lifecycle_scope option lets me ensure that my Lambda can clean up after itself when a destroy operation is executed, and removes the need for a manual Python task every time my team provisions and destroys an Immersion Day.