Health check is too frequently

0

Hello. I run my instance using launching configuration and autoscaling group. On the details tab I see: Health check grace period = 300. Also, I have a load balancer with target group. I set up health checks in the target group: timeout - 5 seconds and interval - 10 seconds. So, I should get one health check request for every 10 seconds. When I visit a cloudwatch log events, I see, that health checks requests sends too frequently, even I filter it by IP's:

2022-04-04T15:08:27.012+03:00
2022-04-04T15:08:31.690+03:00
2022-04-04T15:08:33.899+03:00
2022-04-04T15:08:34.061+03:00
2022-04-04T15:08:43.366+03:00
2022-04-04T15:08:43.898+03:00
2022-04-04T15:08:44.063+03:00
2022-04-04T15:08:45.473+03:00
2022-04-04T15:08:47.303+03:00
2022-04-04T15:08:49.343+03:00
2022-04-04T15:08:53.915+03:00
2022-04-04T15:08:54.067+03:00
2022-04-04T15:08:57.148+03:00
2022-04-04T15:09:02.237+03:00
2022-04-04T15:09:03.913+03:00
2022-04-04T15:09:04.067+03:00
2022-04-04T15:09:13.789+03:00
2022-04-04T15:09:13.920+03:00
2022-04-04T15:09:14.076+03:00
2022-04-04T15:09:15.833+03:00
2022-04-04T15:09:17.460+03:00
2022-04-04T15:09:19.721+03:00

Sometimes I can see 2 health checks in one second. Why? what can I do to fix it?

asked 2 years ago2716 views
2 Answers
0
Accepted Answer

In my case, I need to create different target groups with different health checks configurations and attach them to the load balancer (instead of instance) to listen and forward the request to a specific target group behind the load balancer.

answered 2 years ago
0

Each load balancer node independently checks your targets at the given rate. If you filter the healthcheck requests by just one node's IP, you should see it at a rate of once per 10s.

If the frequency of the healthchecks is causing your application issues, you can either reduce the frequency (which may mean a longer delay before an unhealthy node is detected), or consider creating a specific healthcheck endpoint which the application can use to indicate its status but avoids any other expensive processing your current one may be incurring.

profile pictureAWS
EXPERT
James_S
answered 2 years ago
  • Thank you for your answer. I checked, that even one IP has many health checks record: INFO - 2022-04-05 09:17:25 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.55ms INFO - 2022-04-05 09:17:29 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 0.45ms INFO - 2022-04-05 09:17:32 - tornado.access - 200 GET /userapi/healthcheck (10.0.3.121) 1.11ms I even log into docker of application and using docker logs --follow --since 5m f99ff94a0ff5 check log flow to confirm frequently requests. So my health check works fine, but too frequent. I made next experiment: I have this application with 2 revisions: "green" like active and "blue" like active and alternative. In the alternative "blue" cloudwatch I see correct log flow 1 request per 10 seconds but on the "green" cloudwatch I see frequent log flow like I provided above in this post. So application directly doesn't have an additional health check feature. I checked target group again - it healthy, works good, but I got many health check request which increase total price on my amazon account. Mybe any other idea what I need to check?

  • Also, I found an article on StackOverflow, which promised, that the bug was fixed, but it concerned NLB. I do not use NLB, just load balancer, maybe I need to check the update? any suggestions?

    https://stackoverflow.com/questions/48135471/nlb-target-group-health-checks-are-out-of-control

  • That is strange. I'd suggest opening a support ticket in that case so that the support and service teams can investigate your load balancer to see why it is performing health checks so frequently.

  • James, Support helped me to determine strange behavior: I use the Load balancer with 7 target groups attached to the one EC2 instance. Unfortunately, I do not know how to solve this issue, I can't create each Load Balancer for each target group because I use DNS name for Route 53. Maybe you can provide me with any solution?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions