Wordpress/Nginx Failing under loads with ALB, returning 502.

Question

Hello,

I am testing a Wordpress Instance (PHP8) on Elastic beanstalk setup with a load balancer, running through Nginx. When I visit the website regularly, it works fine and I am able to browse the Wordpress instance.

I wanted to try to perform a very simple load test, so I am using artillery (https://www.artillery.io/) as a load/smoke tester. As soon as the load on the server rises the server starts returning 502 responses. I've taken a look at the logs and I am seeing this.

> ----------------------------------------
/var/log/nginx/access.log
----------------------------------------
10.0.0.20 - - [08/Feb/2022:20:45:56 +0000] "GET / HTTP/1.1" 499 0 "-" "Artillery (https://artillery.io)" "IP-ADDRESS"

> 
10.0.0.20 - - [08/Feb/2022:20:45:57 +0000] "GET / HTTP/1.1" 499 0 "-" "Artillery (https://artillery.io)" "IP-ADDRESS"

The error log says:

> ----------------------------------------
/var/log/nginx/error.log
----------------------------------------
2022/02/08 20:45:44 [error] 2882#2882: *1656 connect() to unix:/run/php-fpm/www.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.0.0.20, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/www.sock:", host: "HOSTNAME.COM"

I've been doing some research and it seems like some people suggest that its a PHP issue, or its an Nginx issue or a Load balancer issue.

I'm not sure where to go with this, I've been trying to debug the issue and try a few different things, has anyone run into this? Does anyone have any debugging tips? Any help/guidance would be appreciated.

SijoJose-AWS · Answer

If this is happening only when doing the load test, then the nginx server unable to respond back to load balancer in timely manner while doing the test. To start with the investigation, you can use any monitoring tools like top/atop/htop to identify if the nginx server resource CPU/memory hitting at 100%. Another resource need to look at it is the disk. You can use "iostat -mydtxz 1" to check the disk usage.

So to start with, open 2 or 3 SSH sessions to the back-end nginx server, and then run top/atop/htop on one session, "iostat -mydtxz 1" on another session, "watch free -m" on another session. Once this is done, run your load test which will hint you at resource utilization.

Sri · Answer

Hello,

I am not suggesting this as an answer, just thought to provide some help

Could you please check the `HTTP Keep-Alive` config in nginx.

As per https://www.tessian.com/blog/how-to-fix-http-502-errors/#:~:text=The%20502%20(Bad%20Gateway)%20status,you%20want%20to%20talk%20to.
```
 The default timeout for the AWS Application Load Balancer is 60 seconds, so we changed the service timeouts to 65 seconds. Barring two hiccoughs shortly after deploying, this has totally fixed it.
```

TimL · Answer

Hello,

If you happen to still be dealing with this, it seems as though load balancing and instance utilization is not the issue as previously mentioned since everything works normally outside of using Artillery. When we really think about the bigger picture of what's happening here, the load test can be comparable to a DDoS attack. I would for sure double check firewall settings.

You had also mentioned PHP as being the potential cause of the issue. To check if PHP-FPM is running, use this command:   **ps aux | grep php-fpm**

If no processes are returned after running that, then simply run the PHP-FPM service and try the test again. PHP could also be timing out and closing before you can connect to a server which would mean increasing the timeout for PHP would be necessary.

Wordpress/Nginx Failing under loads with ALB, returning 502.

/var/log/nginx/access.log

/var/log/nginx/error.log

Relevant content