How do I resolve a failed health check for a load balancer in Amazon EKS?

Short description

When a load balancer health check fails, the load balancer sends a probe to the target group, and the target group forwards the probe to the node or Pod. If the target doesn't respond correctly, then the target appears as unhealthy in the Amazon Elastic Compute Cloud (Amazon EC2) console. Requests to your application might return 502 or 503 errors.

Load balancer health checks in Amazon EKS can fail for the following reasons:

• Pod status issues
• Incorrectly configured label selectors
• Incorrect destination ports
• Security group restrictions
• Network connectivity issues

Resolution

Confirm the target group health status in the Amazon EC2 console

Before you investigate Kubernetes-level configurations, review the target health status and reason code in the Amazon EC2 console. The reason code identifies the specific failure type.

Complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Target Groups.
3. Select your target group, and then choose the Targets tab.
4. Review the Health status column and note the reason code.

For Application Load Balancers, review the reason code to identify your next troubleshooting step:

• For Target.Timeout, confirm that your security group rules allow health check traffic on the target port. If the security groups are correct, then confirm that your application responds within the configured health check timeout.
• For Target.ResponseCodeMismatch, confirm that your application returns an expected HTTP status code on the health check path. Update the success-codes annotation if your application returns non-200 codes.
• For Target.FailedHealthChecks, confirm that your application listens on the target port. If the health check returns a connection refused or reset error, then nothing is listening on that port.

For Network Load Balancers, the console reports only a generic Target.FailedHealthChecks for all failure types. Because Network Load Balancers operate at Layer 4, the console doesn't differentiate between timeout, response code mismatch, or connection refused. Use the manual health check test in this article to identify the root cause for Network Load Balancer targets. For more information, see Health checks for Application Load Balancer target groups.

Confirm the Pod is in Running status

To confirm that the Pod is in Running status and all containers are ready, run the following kubectl get command:

kubectl get pod -n YOUR_NAMESPACE

Note: Replace YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

NAME READY STATUS RESTARTS AGE
podname 1/1 Running 0 16s

If the application container in the Pod's status isn't Running, then the load balancer health check isn't answered and fails.

Confirm that Pod labels match the service selector

For Pod labels, run the following kubectl get command:

kubectl get pod -n YOUR_NAMESPACE --show-labels

Note: Replace YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

NAME READY STATUS RESTARTS AGE LABELS
my-pod 1/1 Running 0 2d19h app=my-app,pod-template-hash=6cc5cd9b9

To confirm that your Kubernetes Service uses the Pod labels, run the following kubectl get command:

kubectl get svc SERVICE_NAME -n YOUR_NAMESPACE -o=jsonpath='{.spec.selector}{""}'

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

{
"app": "alb-instance"
}

Confirm that service endpoints exist

The Kubernetes controller for the service selector continuously scans for Pods that match its selector, and then posts updates to an endpoint object. If you selected an incorrect label, then no endpoint appears.

To review the service configuration and endpoints, run the following kubectl describe command:

kubectl describe svc SERVICE_NAME -n YOUR_NAMESPACE

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

Name: alb-instance
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=alb-instance-1
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: IP_ADDRESS
IPs: IP_ADDRESS
Port: http IP_ADDRESS/TCP
TargetPort: IP_ADDRESS/TCP
NodePort: http 32663/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>

To confirm that the service has registered endpoints, run the following kubectl get command:

kubectl get endpoints SERVICE_NAME -n YOUR_NAMESPACE

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

NAME ENDPOINTS AGE
alb-instance <none> 2d20h

Confirm the service traffic policy and cluster security groups for Application Load Balancers

Unhealthy targets in the Application Load Balancer target groups happen for two reasons:

• You set the spec.externalTrafficPolicy service traffic policy to Local instead of Cluster.
• The node groups in a cluster have different cluster security groups associated with them, and traffic can't flow freely between the node groups.

Note: The node-to-node communication requirement applies specifically when you set externalTrafficPolicy: Cluster.

Confirm the traffic policy setting

To confirm that the traffic policy is correctly configured, run the following kubectl get command:

kubectl get svc SERVICE_NAME -n YOUR_NAMESPACE -o=jsonpath='{.spec.externalTrafficPolicy}{""}'

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

Local

If the setting is incorrect, then run the following kubectl edit command to change the setting to Cluster:

kubectl edit svc SERVICE_NAME -n YOUR_NAMESPACE

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Confirm that cluster security groups allow node-to-node traffic

Complete the following steps:

1. Open the Amazon EC2 console.
2. Choose the healthy instance.
3. Choose the Security tab, and then review the security group ingress rules for traffic from other node groups.
4. Choose the unhealthy instance.
5. Choose the Security tab, and then review the security group ingress rules. If the security group differs from the healthy instance, then choose the security group ID. Choose Edit inbound rules to modify ingress rules. Add inbound rules to allow traffic from the other node groups in the cluster.

Note: If you specify a custom security group on the load balancer by using annotations such as alb.ingress.kubernetes.io/security-groups or service.beta.kubernetes.io/aws-load-balancer-security-groups, then the AWS Load Balancer Controller doesn't auto-manage backend security group rules. In this case, you must manually confirm that the target's security group allows inbound traffic from the load balancer on the health check port. For more information, see alb.ingress.kubernetes.io/security-groups on the Kubernetes SIGs website.

Confirm the targetPort configuration

Your targetPort must match the containerPort in the Pod that the service sends traffic to.

To confirm the port that your targetPort is configured to, run the following kubectl get command:

kubectl get svc SERVICE_NAME -n YOUR_NAMESPACE -o=jsonpath="{.items[*]}{.metadata.name}{'\t'}{.spec.ports[].targetPort}{'\t'}{.spec.ports[].protocol}{''}"

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

alb-instance 8080 TCP

In the example output, the targetPort is configured to port 8080. However, because the containerPort is set to port 80, you must configure the targetPort to 80.

Confirm AWS Load Balancer Controller permissions

The AWS Load Balancer Controller must have the correct permissions to update security groups to allow traffic from the load balancer to instances or Pods. If the controller doesn't have the correct permissions, then you receive errors.

To identify error messages or failed reconciliation events in the AWS Load Balancer Controller deployment logs, run the following kubectl logs command:

kubectl logs deploy/aws-load-balancer-controller -n kube-system

To identify error messages in the individual controller Pod logs, run the following kubectl logs command:

kubectl logs CONTROLLER_POD_NAME -n YOUR_NAMESPACE

Note: Replace CONTROLLER_POD_NAME with your controller Pod name and YOUR_NAMESPACE with your Kubernetes namespace.

Review ingress annotations for Application Load Balancers

To review the Kubernetes ingress annotations for incorrect Application Load Balancer configurations, run the following kubectl describe command:

kubectl describe ing INGRESS_NAME -n YOUR_NAMESPACE

Note: Replace INGRESS_NAME with the name of your Kubernetes Ingress and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

Name: INGRESS_NAME
Namespace: YOUR_NAMESPACE
Address: YOUR_ALB_DNS_NAME
Default backend: SERVICE_NAME:IP_ADDRESS (POD_IP:CONTAINER_PORT)
Rules:
Host Path Backends
---- ---- --------
YOUR_HOSTNAME / SERVICE_NAME:IP_ADDRESS (POD_IP:CONTAINER_PORT)
Annotations: alb.ingress.kubernetes.io/scheme: internet-facing
kubernetes.io/ingress.class: alb
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyReconciled 25m (x7 over 2d21h) ingress Successfully reconciled

Review the following health-check-related annotations to identify incorrect configurations:

• For alb.ingress.kubernetes.io/healthcheck-path, confirm that the path returns a 200 response. If the path doesn't exist, then the target returns a 404 error.
• For alb.ingress.kubernetes.io/healthcheck-port, confirm that this port matches the port where your application listens.
• If the application responds slowly for alb.ingress.kubernetes.io/healthcheck-timeout-seconds, then increase the value.
• For alb.ingress.kubernetes.io/target-type, confirm that the target type matches your deployment. Use ip for Pods directly or instance for NodePort routing.
• For alb.ingress.kubernetes.io/backend-protocol, confirm that the protocol matches what your application expects. For example, use HTTP or HTTPS.
• If your application returns non-200 codes for health checks for alb.ingress.kubernetes.io/success-codes, then add the expected codes.

To find ingress annotations that are specific to your use case, see Ingress annotations on the Kubernetes website.

Review Kubernetes Service annotations for Network Load Balancers

To review the Kubernetes Service annotations for incorrect Network Load Balancer configurations, run the following kubectl describe command:

kubectl describe svc SERVICE_NAME -n YOUR_NAMESPACE

Note: Replace SERVICE_NAME with your Kubernetes Service and YOUR_NAMESPACE with your Kubernetes namespace.

Example output:

Name: nlb-ip
Namespace: default
Labels: <none>
Annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-type: external
Selector: app=nlb-ip
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: IP_ADDRESS
IPs: IP_ADDRESS
LoadBalancer Ingress: NLB_DNS_NAME
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32663/TCP
Endpoints: POD_IP
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>

Note: Note the value of APPLICATION_POD_IP to run a health check command in a later step.

To find Kubernetes Service annotations for your use case, see Service annotations on the Kubernetes website.

Manually test a health check

To retrieve your application Pod IP address, run the following kubectl get command:

kubectl get pod -n YOUR_NAMESPACE -o wide

Note: Replace YOUR_NAMESPACE with your Kubernetes namespace.

To run a test Pod that manually tests a health check within the cluster, run the following kubectl run command:

kubectl run -n YOUR_NAMESPACE troubleshoot -it --rm --image=amazonlinux -- /bin/bash

Note: Replace YOUR_NAMESPACE with your Kubernetes namespace.

To run the HTTP health check, run the following curl command:

curl -Iv APPLICATION_POD_IP/HEALTH_CHECK_PATH

Note: Replace APPLICATION_POD_IP with your application Pod IP address and HEALTH_CHECK_PATH with the health check path for your Application Load Balancer target group.

Example command:

curl -Iv APPLICATION_POD_IP

Example output:

* Trying APPLICATION_POD_IP:80...
* Connected to APPLICATION_POD_IP (APPLICATION_POD_IP) port 80 (#0)
> HEAD / HTTP/1.1
> Host: APPLICATION_POD_IP
> User-Agent: curl/7.78.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.21.3
Server: nginx/1.21.3
< Date: Tue, 26 Oct 2021 05:10:17 GMT
Date: Tue, 26 Oct 2021 05:10:17 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 615
Content-Length: 615
< Last-Modified: Tue, 07 Sep 2021 15:21:03 GMT
Last-Modified: Tue, 07 Sep 2021 15:21:03 GMT
< Connection: keep-alive
Connection: keep-alive
< ETag: "example-etag-value"
ETag: "example-etag-value"
< Accept-Ranges: bytes
Accept-Ranges: bytes
<
* Connection #0 to host APPLICATION_POD_IP left intact

Review the HTTP response status code. If the response status code is 200 OK, then your application correctly responds to the health check path.

If the HTTP response status code is 3xx or 4xx, then change your health check path. The following annotation responds with 200 OK:

alb.ingress.kubernetes.io/healthcheck-path: /ping

-or-

Use the following annotation on the ingress resource to add a successful health check response status code range:

alb.ingress.kubernetes.io/success-codes: 200-399

For TCP health checks, run the following command to install the netcat command:

yum update -y && yum install -y nc

To test the TCP health checks, run the following command:

nc -z -v APPLICATION_POD_IP CONTAINER_PORT_NUMBER

Note: Replace APPLICATION_POD_IP with your application Pod IP address and CONTAINER_PORT_NUMBER with your container port.

Example command:

nc -z -v APPLICATION_POD_IP IP_ADDRESS

Example output:

Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to APPLICATION_POD_IP:IP_ADDRESS.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

Confirm network connectivity between the load balancer and targets

For networking issues, confirm that the following configurations allow traffic between the load balancer and targets:

• The node groups in your Amazon EKS cluster can freely communicate with each other.
• The network access control list (network ACL) that's associated with the subnet where your Pods run allows traffic from the load balancer subnet CIDR range.
• The network ACL that's associated with your load balancer subnet allows return traffic on the ephemeral port range from the subnet where the Pods run.
• The route table allows local traffic from within the virtual private cloud (VPC) CIDR range.

Restart the kube-proxy

If the kube-proxy that runs on each node doesn't work correctly, then the kube-proxy might fail to update the iptables rules for the service and endpoints.

To restart the kube-proxy to force it to recheck and update iptables rules, run the following command:

kubectl rollout restart daemonset.apps/kube-proxy -n kube-system

Example output:

daemonset.apps/kube-proxy restarted

Related information

How do I use AWS Load Balancer Controller to set up an Application Load Balancer on an Amazon EC2 node group in Amazon EKS?

How do I troubleshoot load balancers created by the Kubernetes service controller in Amazon EKS?

How do I automatically discover the subnets that my Application Load Balancer uses in Amazon EKS?