Health check is too frequently

0

Hello. I run my instance using launching configuration and autoscaling group. On the details tab I see: Health check grace period = 300. Also, I have a load balancer with target group. I set up health checks in the target group: timeout - 5 seconds and interval - 10 seconds. So, I should get one health check request for every 10 seconds. When I visit a cloudwatch log events, I see, that health checks requests sends too frequently, even I filter it by IP's:

2022-04-04T15:08:27.012+03:00
2022-04-04T15:08:31.690+03:00
2022-04-04T15:08:33.899+03:00
2022-04-04T15:08:34.061+03:00
2022-04-04T15:08:43.366+03:00
2022-04-04T15:08:43.898+03:00
2022-04-04T15:08:44.063+03:00
2022-04-04T15:08:45.473+03:00
2022-04-04T15:08:47.303+03:00
2022-04-04T15:08:49.343+03:00
2022-04-04T15:08:53.915+03:00
2022-04-04T15:08:54.067+03:00
2022-04-04T15:08:57.148+03:00
2022-04-04T15:09:02.237+03:00
2022-04-04T15:09:03.913+03:00
2022-04-04T15:09:04.067+03:00
2022-04-04T15:09:13.789+03:00
2022-04-04T15:09:13.920+03:00
2022-04-04T15:09:14.076+03:00
2022-04-04T15:09:15.833+03:00
2022-04-04T15:09:17.460+03:00
2022-04-04T15:09:19.721+03:00

Sometimes I can see 2 health checks in one second. Why? what can I do to fix it?

posta 2 anni fa2806 visualizzazioni
2 Risposte
0
Risposta accettata

In my case, I need to create different target groups with different health checks configurations and attach them to the load balancer (instead of instance) to listen and forward the request to a specific target group behind the load balancer.

con risposta 2 anni fa
profile picture
ESPERTO
verificato 18 ore fa
0

Each load balancer node independently checks your targets at the given rate. If you filter the healthcheck requests by just one node's IP, you should see it at a rate of once per 10s.

If the frequency of the healthchecks is causing your application issues, you can either reduce the frequency (which may mean a longer delay before an unhealthy node is detected), or consider creating a specific healthcheck endpoint which the application can use to indicate its status but avoids any other expensive processing your current one may be incurring.

profile pictureAWS
ESPERTO
James_S
con risposta 2 anni fa
  • Thank you for your answer. I checked, that even one IP has many health checks record: INFO - 2022-04-05 09:17:25 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.55ms INFO - 2022-04-05 09:17:29 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.45ms INFO - 2022-04-05 09:17:32 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 1.11ms I even log into docker of application and using docker logs --follow --since 5m f99ff94a0ff5 check log flow to confirm frequently requests. So my health check works fine, but too frequent. I made next experiment: I have this application with 2 revisions: "green" like active and "blue" like active and alternative. In the alternative "blue" cloudwatch I see correct log flow 1 request per 10 seconds but on the "green" cloudwatch I see frequent log flow like I provided above in this post. So application directly doesn't have an additional health check feature. I checked target group again - it healthy, works good, but I got many health check request which increase total price on my amazon account. Mybe any other idea what I need to check?

  • Also, I found an article on StackOverflow, which promised, that the bug was fixed, but it concerned NLB. I do not use NLB, just load balancer, maybe I need to check the update? any suggestions?

    https://stackoverflow.com/questions/48135471/nlb-target-group-health-checks-are-out-of-control

  • That is strange. I'd suggest opening a support ticket in that case so that the support and service teams can investigate your load balancer to see why it is performing health checks so frequently.

  • James, Support helped me to determine strange behavior: I use the Load balancer with 7 target groups attached to the one EC2 instance. Unfortunately, I do not know how to solve this issue, I can't create each Load Balancer for each target group because I use DNS name for Route 53. Maybe you can provide me with any solution?

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande