Why the Target group is empty when I create a NLB point to a custom end-point

When you create a Kubernetes Service of type 'LoadBalancer', the Network Load Balancer (NLB) is provisioned and assigned an external IP address. The NLB forwards traffic to the backend Pods based on the rules defined in the Service.

In this case, a custom endpoint object that has a different IP address from the Pod is being used. This custom endpoint is likely a separate resource in your cluster that is not directly associated with the pods

When you install the Helm chart in a single step, which includes both the Pod and the Service with the endpoint, it's possible that the NLB starts directing traffic to the Pod before the custom endpoint is fully ready and registered as a target.

However, when you install them in two steps, first the Pod and then the Service with the endpoint, it provides more time for the custom endpoint to become fully available and registered as a target in the target group of the NLB.

To ensure that the custom endpoint is registered as a target before traffic is directed to the Pods, you can try the following approaches:

Add readiness probes to your Pod: By specifying a readiness probe in the Pod's configuration, you can delay the Pod's readiness until the custom endpoint is fully available. This will prevent traffic from being directed to the Pod before the endpoint is ready [1].

Use an initializer or a sidecar container: You can create an initializer or a sidecar container that waits for the custom endpoint to become available before allowing traffic to the Pod. The initializer/sidecar can periodically check the readiness of the endpoint and update a status that the Pod's readiness probe can monitor.

Use an operator or a custom controller: You can develop a custom operator or controller that ensures the proper synchronization between the Pod, the custom endpoint, and the Service with the NLB. The operator/controller can handle the ordering and dependencies between the resources to ensure that the NLB only starts directing traffic once the endpoint is registered.

By employing one of these approaches, you can ensure that the custom endpoint is fully registered as a target before traffic is directed to the Pods, even when deploying everything in a single step using Helm.