Elastic Beanstalk Traffic Spike Caused HTTP 504

Question

Hello,

Recently, we had an unexpected traffic spike of roughly 4x our average inbound network, causing the CPU utilization of our instance to jump over 40%. The instance was unable to recover and resulted in 504 errors until it could be manually restarted. The server logs aren't yielding any helpful information.

1. Is there any way to find out in more detail what specifically caused the instance to crap out?
2. How do we prevent an outage of this kind in the future? We have since adjusted our auto-scaling thresholds to a CPU utilization of 35%, but we're not convinced that's going to be bulletproof. Is there a better way to check for unhealthy instances and automatically spin up a new one.

Thanks in advance!

Answer

Firstly a comment, 35% CPU utilization is very low - and by itself should not cause the application to crash - so I suspect that something else is causing your application to fail to respond, so look at things like the number of concurrent threads your application web server can service and so on.

Then to diagnose what is actually happening, you can instrument the instance with something like SAR, examine web-server logs and your application server logs. You may need to increase verbosity in these logs.

Also - if this has happened recently - and your have AWS Support - open a support case - support can generally provide you with helpful guidance on troubleshooting and configuring your operating system and scaling.

Elastic Beanstalk Traffic Spike Caused HTTP 504

Relevanter Inhalt