Health check is too frequently

0

Hello. I run my instance using launching configuration and autoscaling group. On the details tab I see: Health check grace period = 300. Also, I have a load balancer with target group. I set up health checks in the target group: timeout - 5 seconds and interval - 10 seconds. So, I should get one health check request for every 10 seconds. When I visit a cloudwatch log events, I see, that health checks requests sends too frequently, even I filter it by IP's:

2022-04-04T15:08:27.012+03:00
2022-04-04T15:08:31.690+03:00
2022-04-04T15:08:33.899+03:00
2022-04-04T15:08:34.061+03:00
2022-04-04T15:08:43.366+03:00
2022-04-04T15:08:43.898+03:00
2022-04-04T15:08:44.063+03:00
2022-04-04T15:08:45.473+03:00
2022-04-04T15:08:47.303+03:00
2022-04-04T15:08:49.343+03:00
2022-04-04T15:08:53.915+03:00
2022-04-04T15:08:54.067+03:00
2022-04-04T15:08:57.148+03:00
2022-04-04T15:09:02.237+03:00
2022-04-04T15:09:03.913+03:00
2022-04-04T15:09:04.067+03:00
2022-04-04T15:09:13.789+03:00
2022-04-04T15:09:13.920+03:00
2022-04-04T15:09:14.076+03:00
2022-04-04T15:09:15.833+03:00
2022-04-04T15:09:17.460+03:00
2022-04-04T15:09:19.721+03:00

Sometimes I can see 2 health checks in one second. Why? what can I do to fix it?

gefragt vor 2 Jahren2803 Aufrufe
2 Antworten
0
Akzeptierte Antwort

In my case, I need to create different target groups with different health checks configurations and attach them to the load balancer (instead of instance) to listen and forward the request to a specific target group behind the load balancer.

beantwortet vor 2 Jahren
profile picture
EXPERTE
überprüft vor 8 Stunden
0

Each load balancer node independently checks your targets at the given rate. If you filter the healthcheck requests by just one node's IP, you should see it at a rate of once per 10s.

If the frequency of the healthchecks is causing your application issues, you can either reduce the frequency (which may mean a longer delay before an unhealthy node is detected), or consider creating a specific healthcheck endpoint which the application can use to indicate its status but avoids any other expensive processing your current one may be incurring.

profile pictureAWS
EXPERTE
James_S
beantwortet vor 2 Jahren
  • Thank you for your answer. I checked, that even one IP has many health checks record: INFO - 2022-04-05 09:17:25 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.55ms INFO - 2022-04-05 09:17:29 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.45ms INFO - 2022-04-05 09:17:32 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 1.11ms I even log into docker of application and using docker logs --follow --since 5m f99ff94a0ff5 check log flow to confirm frequently requests. So my health check works fine, but too frequent. I made next experiment: I have this application with 2 revisions: "green" like active and "blue" like active and alternative. In the alternative "blue" cloudwatch I see correct log flow 1 request per 10 seconds but on the "green" cloudwatch I see frequent log flow like I provided above in this post. So application directly doesn't have an additional health check feature. I checked target group again - it healthy, works good, but I got many health check request which increase total price on my amazon account. Mybe any other idea what I need to check?

  • Also, I found an article on StackOverflow, which promised, that the bug was fixed, but it concerned NLB. I do not use NLB, just load balancer, maybe I need to check the update? any suggestions?

    https://stackoverflow.com/questions/48135471/nlb-target-group-health-checks-are-out-of-control

  • That is strange. I'd suggest opening a support ticket in that case so that the support and service teams can investigate your load balancer to see why it is performing health checks so frequently.

  • James, Support helped me to determine strange behavior: I use the Load balancer with 7 target groups attached to the one EC2 instance. Unfortunately, I do not know how to solve this issue, I can't create each Load Balancer for each target group because I use DNS name for Route 53. Maybe you can provide me with any solution?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen