Skip to content

Do Kubernetes LoadBalancer services need to be deleted before terraform destroy when EKS is provisioned by Terraform but NLBs are not?

0

I'm using Terraform to provision an EKS cluster, but the NLBs are created implicitly by Kubernetes when I deploy Service resources of type LoadBalancer. Terraform has no knowledge of these NLBs.

My concern is that running terraform destroy might fail or leave orphaned resources because the VPC/subnets still have active NLBs that Terraform didn't create. In fact I have a reason to doubt it as I explained here. Other resources I have looked at and believe are making my point even clearer:

The AWS EKS docs suggest deleting LoadBalancer services before deleting the cluster, and aws elbv2 wait load-balancers-deleted seems like the right primitive to confirm cleanup. But here I am not the one who wanted to do this since I am big on IaC reigns supreme. I also have read questions like this which again hints the same thing.

Questions:

  1. Will terraform destroy actually fail in this scenario, or does EKS handle NLB cleanup automatically on cluster deletion?
  2. If manual cleanup is needed, is my cleanup hook the idiomatic approach you guys use daily in your professional projects?
  3. Does the answer differ between EKS Auto Mode  and the self-managed AWS Load Balancer Controller?
  4. How would you approach this? Maybe I should have not go with auto mode and instead managed it myself which is kinda not really OK either since then we are or Terraform is meddling in what is Kubernetes for. Am I wrong about this?
3 Answers
1

When you provision Amazon EKS with Terraform, the cluster itself is managed by Terraform, but LoadBalancer Services (NLBs/ALBs) created by Kubernetes are not tracked in Terraform state. That means Terraform has no knowledge of them, and terraform destroy will not automatically clean them up.

https://docs.aws.amazon.com/prescriptive-guidance/latest/getting-started-terraform/providers.html

EXPERT

answered a month ago

EXPERT

reviewed a month ago

0

Yes, Kubernetes LoadBalancer services should be deleted before running terraform destroy on an EKS cluster, even when Terraform didn't create the load balancers directly. Here's why and how to handle it:

1. Will terraform destroy fail?

Yes, it can fail or leave orphaned resources. When you delete an EKS cluster without first removing services with external load balancers, you risk having orphaned resources in your VPC that prevent proper cleanup. The load balancers created by Kubernetes exist outside Terraform's state, so Terraform cannot manage their deletion. If these load balancers still reference VPC resources like subnets or security groups that Terraform is trying to destroy, the destroy operation will fail with dependency errors.

2. Is a cleanup hook the right approach?

Yes, your cleanup hook approach is a valid and commonly used pattern. Before deleting the cluster, you need to:

  • List all services across all namespaces to identify those with EXTERNAL-IP values
  • Delete those services using kubectl to properly release the load balancers
  • Delete any ingress resources, as application load balancers will remain even after cluster deletion if ingresses aren't removed

This ensures that Kubernetes properly cleans up the AWS resources it created before Terraform attempts to destroy the underlying infrastructure. Many teams implement this as a pre-destroy provisioner or external script that runs before terraform destroy.

3. Does EKS Auto Mode differ?

The fundamental requirement remains the same regardless of whether you're using EKS Auto Mode or a self-managed AWS Load Balancer Controller. In both cases, load balancers created through Kubernetes Service resources must be deleted before cluster deletion to avoid orphaned resources. EKS Auto Mode does handle the deletion of its own managed nodes and some infrastructure, but it doesn't automatically clean up load balancers created by your application's Service resources.

4. Recommended approach:

Your instinct about Infrastructure as Code is correct, but there's a natural boundary here. Terraform manages the cluster infrastructure, while Kubernetes manages application-level resources including load balancers created through Service definitions. This separation is actually appropriate.

The idiomatic approach is:

  • Use Terraform for cluster infrastructure (VPC, EKS cluster, node groups)
  • Use Kubernetes manifests or Helm for application resources (Services, Ingresses)
  • Implement a cleanup step that deletes Kubernetes resources with external dependencies before running terraform destroy

Your cleanup hook that runs kubectl commands to delete services and ingresses before cluster destruction is the right pattern. This respects the separation of concerns while ensuring proper cleanup order. Many production environments use similar approaches, often wrapped in CI/CD pipelines that orchestrate the proper sequence of operations.
Sources
DeleteCluster - Amazon EKS
Delete a cluster - Amazon EKS

answered a month ago

EXPERT

reviewed a month ago

0

The answer to question 1 depends on question 3, so starting there.

Auto Mode vs the self-managed AWS Load Balancer Controller

They are documented differently:

  • Self-managed controller: the EKS delete cluster guide says to delete every Service with an EXTERNAL-IP (and every Ingress) before deleting the cluster, and that an ALB from an Ingress you did not delete "remains even if you deleted the cluster".
  • Auto Mode: the same guide, under "Considerations for EKS Auto Mode", says that when the cluster is deleted "All load balancers will be deleted", along with the Auto Mode nodes. The Disable EKS Auto Mode page words it as "EKS will delete all Load Balancers operated by EKS Auto Mode", and adds that EBS volumes provisioned by Auto Mode are not deleted.

So on Auto Mode, with Services using the default class eks.amazonaws.com/nlb, the NLBs should not be orphaned forever. That is not the same as terraform destroy succeeding in one run, though.

Why destroy can still fail

Terraform deletes the cluster and then moves straight on to subnets, security groups and the VPC. If an NLB or its network interfaces are still in those subnets at that moment, the VPC side fails with a dependency error, whoever cleans the NLB up later. I would not bet a pipeline on the timing, and the leftover EBS volumes from PVCs are yours to delete either way.

What I would do

Make Kubernetes remove its own load balancers while the cluster and the controller still exist, then destroy the infrastructure. Two ways:

  1. Keep the Services in Terraform state (kubernetes or helm provider) with a dependency on the cluster and node resources. Terraform destroys in reverse dependency order, so the Services go first, while the controller is still running to act on them.
  2. Or a pre-destroy step, which is what your hook does:
kubectl delete ingress --all -A
kubectl get svc -A -o json \
  | jq -r '.items[] | select(.spec.type=="LoadBalancer") | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read ns name; do kubectl delete svc -n "$ns" "$name"; done

then wait until the load balancers are actually gone before terraform destroy. aws elbv2 wait load-balancers-deleted works if you record the ARNs first.

Either way the order is the fix: application resources that create AWS resources go first, cluster second, network last. Terraform owning the cluster and Kubernetes owning the load balancers is a fine split, as long as the destroy respects that order.

Sources:

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.