AWS Global Accelerator site health check opens many connections on apache server with status Reading Request

0

This situation occurred on August 13 at 23:30 UTC

I use AWS Global Accelerator in the oregon us-west-2 region for users to access the application, the application is on an apache2 server running on ubuntu.

The global accelerator endpoint points directly to the EC2 instances. All EC2 instances associated with the endpoint, since the indicated time, presented an anomaly in the number of connections in the Reading Request state.

This high number of connections came from the range 15.177.0.0/18 which are the Amazon IPs to perform site health checks: ROUTE53_HEALTHCHECKS

I was able to identify these IPs, since I applied the mod_antiloris module on my servers and the following errors appeared in the logs (the time of the logs is in ECT)

And even if I applied mod_antiloris and set the number of connections allowed for reading to 2, as the health check makes many connections from many IPs to the servers, the number of connections still increased, it's like a DDoS attack.

How ​​can I solve the problem in Apache or how should I configure global accelerator so that it does not open so many connections?

Sample Logs.

[Wed Aug 14 00:38:08.060297 2024] [antiloris:warn] [pid 24539:tid 140602314282752] [client 15.177.6.197:31986] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:15.775106 2024] [antiloris:warn] [pid 24539:tid 140602314282752] [client 15.177.42.234:9956] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:18.054674 2024] [antiloris:warn] [pid 24538:tid 140602314282752] [client 15.177.46.71:56746] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:18.584646 2024] [antiloris:warn] [pid 26768:tid 140602480871168] [client 15.177.42.177:63530] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:19.319311 2024] [antiloris:warn] [pid 26768:tid 140602480871168] [client 15.177.46.166:23534] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:19.786816 2024] [antiloris:warn] [pid 24539:tid 140602314282752] [client 15.177.34.171:63152] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:21.176524 2024] [antiloris:warn] [pid 24538:tid 140602314282752] [client 15.177.38.160:25022] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:22.199159 2024] [antiloris:warn] [pid 26972:tid 140601844553472] [client 15.177.58.79:18504] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:22.522345 2024] [antiloris:warn] [pid 26972:tid 140601836160768] [client 15.177.62.171:58800] Connection rejected by Antiloris, too many connections
[Wed Aug 14 00:38:22.763295 2024] [antiloris:warn] [pid 24538:tid 140602480871168] [client 15.177.34.188:40872] Connection rejected by Antiloris, too many connections

this image show the stats of apache workers. suddenly the workers in reading status increase. this is cause by health check process in aws global accelerator

Apache workers

asked 2 months ago81 views
2 Answers
0

Hi,

Have a look at https://docs.aws.amazon.com/global-accelerator/latest/dg/introduction-get-started.html

Would configuration of traffic dial percentage be solution to your problem ?

Configure regional endpoint groups for your accelerator: You can select one or 
more regional endpoint groups to add to your listener. The listener routes requests 
to the endpoints that you've added to an endpoint group.

For a standard accelerator, Global Accelerator monitors the health of endpoints within 
the group by using the health check settings that are defined for each of your endpoints. 
For each endpoint group in a standard accelerator, you can configure a traffic dial percentage 
to control the percentage of traffic that an endpoint group will accept. The percentage is 
applied only to traffic that is already directed to the endpoint group, not all listener traffic. By 
default, the traffic dial is set to 100% for all regional endpoint groups.

For a custom routing accelerators, traffic is deterministically routed to a specific destination in 
a VPC subnet, based on the listener port that the traffic is received on.

Best,

Diidier

profile pictureAWS
EXPERT
answered 2 months ago
  • Thanks for your answer, but this isn't the problem.

    The HEALTH CHECKS process create too many connections to all the servers behind the endpoint in global accelerator.

    When the connections are created, this leave open in status "Reading Request" in my apaches servers. this is the problem

0

According to the documentation:

After each health check is completed, the listener closes the connection that was established for the health check.

This contradicts what you see in your environment.

I suggest you take traffic capture on your apache server using tools like tcpdump to verify the actual behavior (e.g., that after the TCP handshake is established Global Accelerator sends or doesn't send FIN or RST to close the connection).

profile pictureAWS
EXPERT
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions