Wordpress/Nginx Failing under loads with ALB, returning 502.

0

Hello,

I am testing a Wordpress Instance (PHP8) on Elastic beanstalk setup with a load balancer, running through Nginx. When I visit the website regularly, it works fine and I am able to browse the Wordpress instance.

I wanted to try to perform a very simple load test, so I am using artillery (https://www.artillery.io/) as a load/smoke tester. As soon as the load on the server rises the server starts returning 502 responses. I've taken a look at the logs and I am seeing this.


/var/log/nginx/access.log

10.0.0.20 - - [08/Feb/2022:20:45:56 +0000] "GET / HTTP/1.1" 499 0 "-" "Artillery (https://artillery.io)" "IP-ADDRESS"

10.0.0.20 - - [08/Feb/2022:20:45:57 +0000] "GET / HTTP/1.1" 499 0 "-" "Artillery (https://artillery.io)" "IP-ADDRESS"

The error log says:


/var/log/nginx/error.log

2022/02/08 20:45:44 [error] 2882#2882: *1656 connect() to unix:/run/php-fpm/www.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.0.0.20, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/www.sock:", host: "HOSTNAME.COM"

I've been doing some research and it seems like some people suggest that its a PHP issue, or its an Nginx issue or a Load balancer issue.

I'm not sure where to go with this, I've been trying to debug the issue and try a few different things, has anyone run into this? Does anyone have any debugging tips? Any help/guidance would be appreciated.

Rei
asked 2 years ago936 views
3 Answers
1

If this is happening only when doing the load test, then the nginx server unable to respond back to load balancer in timely manner while doing the test. To start with the investigation, you can use any monitoring tools like top/atop/htop to identify if the nginx server resource CPU/memory hitting at 100%. Another resource need to look at it is the disk. You can use "iostat -mydtxz 1" to check the disk usage.

So to start with, open 2 or 3 SSH sessions to the back-end nginx server, and then run top/atop/htop on one session, "iostat -mydtxz 1" on another session, "watch free -m" on another session. Once this is done, run your load test which will hint you at resource utilization.

answered 2 years ago
  • in HTOP The memory never goes over 1 gig, the system has 8 gigs (Using an M5.Large EC2 on EBS). The CPU never goes above 10% and the iostat looks clean too? But I'm not sure if I'm reading it right but %util was either 0 or blank, so I don't think its because resources are being taxed.

0

Hello,

I am not suggesting this as an answer, just thought to provide some help

Could you please check the HTTP Keep-Alive config in nginx.

As per https://www.tessian.com/blog/how-to-fix-http-502-errors/#:~:text=The%20502%20(Bad%20Gateway)%20status,you%20want%20to%20talk%20to.

 The default timeout for the AWS Application Load Balancer is 60 seconds, so we changed the service timeouts to 65 seconds. Barring two hiccoughs shortly after deploying, this has totally fixed it.
profile picture
Sri
answered 2 years ago
  • I set this in nginx.conf

    http { keepalive_time 1h keepalive_timeout 90s; }

    Still getting 502

0

Hello,

If you happen to still be dealing with this, it seems as though load balancing and instance utilization is not the issue as previously mentioned since everything works normally outside of using Artillery. When we really think about the bigger picture of what's happening here, the load test can be comparable to a DDoS attack. I would for sure double check firewall settings.

You had also mentioned PHP as being the potential cause of the issue. To check if PHP-FPM is running, use this command: ps aux | grep php-fpm

If no processes are returned after running that, then simply run the PHP-FPM service and try the test again. PHP could also be timing out and closing before you can connect to a server which would mean increasing the timeout for PHP would be necessary.

TimL
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions