Whenever we get large traffic spikes on our Rails 4 app, it seems passenger can't handle the load and starts showing the dreaded This website is under heavy load (queue full) error page.
I'm trying to optimize things as much as i can on the app side, but looking at the status of passenger when this occurs, seems to me a lot of the processes are just stuck:
sudo passenger-status
Version : 5.3.5
Date : 2021-03-18 21:59:52 +0000
Instance: xGyoOAEC (nginx/1.18.0 Phusion_Passenger/5.3.5)
----------- General information -----------
Max pool size : 160
App groups : 2
Processes : 160
Requests in top-level queue : 0
----------- Application groups -----------
/var/www/app1 (production):
App root: /var/www/app1
Requests in queue: 0
* PID: 18014 Sessions: 1 Processed: 113 Uptime: 9m 21s
CPU: 0% Memory : 170M Last used: 1s ago
* PID: 18076 Sessions: 0 Processed: 12 Uptime: 9m 19s
CPU: 0% Memory : 131M Last used: 1m 44s ago
/var/www/app2 (production):
App root: /var/www/app2
Requests in queue: 250
* PID: 17786 Sessions: 1 Processed: 13 Uptime: 9m 24s
CPU: 0% Memory : 85M Last used: 1m 24s ago
* PID: 17833 Sessions: 1 Processed: 23 Uptime: 9m 22s
CPU: 0% Memory : 119M Last used: 1m 22s ago
* PID: 17978 Sessions: 1 Processed: 16 Uptime: 9m 21s
CPU: 0% Memory : 118M Last used: 1m 20s ago
* PID: 18122 Sessions: 1 Processed: 16 Uptime: 9m 19s
CPU: 0% Memory : 102M Last used: 1m 18s ago
* PID: 18183 Sessions: 1 Processed: 11 Uptime: 9m 17s
CPU: 0% Memory : 95M Last used: 1m 16s ago
* PID: 18233 Sessions: 1 Processed: 5 Uptime: 9m 15s
CPU: 0% Memory : 117M Last used: 1m 14s ago
* PID: 18289 Sessions: 1 Processed: 14 Uptime: 9m 13s
CPU: 0% Memory : 124M Last used: 3m 13s ago
* PID: 18351 Sessions: 1 Processed: 18 Uptime: 9m 11s
CPU: 0% Memory : 96M Last used: 1m 11s ago
* PID: 18411 Sessions: 1 Processed: 24 Uptime: 9m 10s
CPU: 0% Memory : 97M Last used: 1m 9s ago
* PID: 18463 Sessions: 1 Processed: 14 Uptime: 9m 8s
CPU: 0% Memory : 98M Last used: 1m 7s ago
* PID: 18516 Sessions: 1 Processed: 29 Uptime: 9m 6s
CPU: 0% Memory : 98M Last used: 1m 5s ago
* PID: 18575 Sessions: 1 Processed: 15 Uptime: 9m 4s
CPU: 0% Memory : 95M Last used: 1m 4s ago
* PID: 18623 Sessions: 1 Processed: 22 Uptime: 9m 2s
CPU: 0% Memory : 100M Last used: 1m 2s ago
* PID: 18673 Sessions: 1 Processed: 32 Uptime: 9m 1s
CPU: 0% Memory : 99M Last used: 1m 0s ago
* PID: 18729 Sessions: 1 Processed: 15 Uptime: 8m 59s
CPU: 0% Memory : 86M Last used: 58s ago
* PID: 18788 Sessions: 1 Processed: 19 Uptime: 8m 57s
CPU: 0% Memory : 96M Last used: 56s ago
* PID: 18839 Sessions: 1 Processed: 29 Uptime: 8m 55s
CPU: 0% Memory : 98M Last used: 54s ago
* PID: 18895 Sessions: 1 Processed: 18 Uptime: 8m 53s
CPU: 0% Memory : 85M Last used: 53s ago
* PID: 18944 Sessions: 1 Processed: 19 Uptime: 8m 51s
CPU: 0% Memory : 103M Last used: 51s ago
* PID: 18998 Sessions: 1 Processed: 10 Uptime: 8m 50s
CPU: 0% Memory : 97M Last used: 2m 49s ago
* PID: 19061 Sessions: 1 Processed: 32 Uptime: 8m 48s
CPU: 0% Memory : 98M Last used: 47s ago
* PID: 19125 Sessions: 1 Processed: 17 Uptime: 8m 46s
CPU: 0% Memory : 99M Last used: 45s ago
* PID: 19183 Sessions: 1 Processed: 10 Uptime: 8m 44s
CPU: 0% Memory : 118M Last used: 43s ago
* PID: 19232 Sessions: 1 Processed: 8 Uptime: 8m 42s
CPU: 0% Memory : 98M Last used: 42s ago
* PID: 19286 Sessions: 1 Processed: 36 Uptime: 8m 41s
CPU: 0% Memory : 118M Last used: 40s ago
* PID: 19342 Sessions: 1 Processed: 16 Uptime: 8m 39s
CPU: 0% Memory : 118M Last used: 38s ago
* PID: 19395 Sessions: 1 Processed: 12 Uptime: 8m 37s
CPU: 0% Memory : 98M Last used: 36s ago
* PID: 19445 Sessions: 1 Processed: 11 Uptime: 8m 35s
CPU: 0% Memory : 95M Last used: 2m 35s ago
* PID: 19504 Sessions: 1 Processed: 12 Uptime: 8m 33s
CPU: 0% Memory : 103M Last used: 33s ago
* PID: 19557 Sessions: 1 Processed: 37 Uptime: 8m 31s
CPU: 0% Memory : 102M Last used: 31s ago
* PID: 19608 Sessions: 1 Processed: 11 Uptime: 8m 30s
CPU: 0% Memory : 116M Last used: 29s ago
* PID: 19664 Sessions: 1 Processed: 14 Uptime: 8m 28s
CPU: 0% Memory : 119M Last used: 27s ago
* PID: 19712 Sessions: 1 Processed: 19 Uptime: 8m 26s
CPU: 0% Memory : 97M Last used: 25s ago
* PID: 19774 Sessions: 1 Processed: 22 Uptime: 8m 24s
CPU: 0% Memory : 100M Last used: 23s ago
* PID: 19824 Sessions: 1 Processed: 19 Uptime: 8m 22s
CPU: 0% Memory : 115M Last used: 21s ago
* PID: 19954 Sessions: 1 Processed: 12 Uptime: 8m 20s
CPU: 0% Memory : 95M Last used: 20s ago
* PID: 20021 Sessions: 1 Processed: 4 Uptime: 8m 19s
CPU: 0% Memory : 117M Last used: 18s ago
* PID: 20073 Sessions: 1 Processed: 16 Uptime: 8m 17s
CPU: 0% Memory : 96M Last used: 16s ago
* PID: 20124 Sessions: 1 Processed: 11 Uptime: 8m 15s
CPU: 0% Memory : 95M Last used: 14s ago
* PID: 20182 Sessions: 1 Processed: 19 Uptime: 8m 13s
CPU: 0% Memory : 106M Last used: 13s ago
* PID: 20236 Sessions: 1 Processed: 8 Uptime: 8m 11s
CPU: 0% Memory : 100M Last used: 11s ago
* PID: 20287 Sessions: 1 Processed: 16 Uptime: 8m 10s
CPU: 0% Memory : 98M Last used: 9s ago
* PID: 20344 Sessions: 1 Processed: 7 Uptime: 8m 8s
CPU: 0% Memory : 100M Last used: 7s ago
* PID: 20397 Sessions: 1 Processed: 17 Uptime: 8m 6s
CPU: 0% Memory : 96M Last used: 5s ago
* PID: 20456 Sessions: 1 Processed: 11 Uptime: 8m 4s
CPU: 0% Memory : 118M Last used: 3s ago
* PID: 20510 Sessions: 1 Processed: 16 Uptime: 8m 2s
CPU: 0% Memory : 117M Last used: 2s ago
* PID: 20565 Sessions: 1 Processed: 6 Uptime: 8m 1s
CPU: 0% Memory : 97M Last used: 2m 0s ago
* PID: 20622 Sessions: 1 Processed: 8 Uptime: 7m 59s
CPU: 0% Memory : 94M Last used: 1m 58s ago
* PID: 20674 Sessions: 1 Processed: 9 Uptime: 7m 57s
CPU: 0% Memory : 120M Last used: 1m 56s ago
* PID: 20726 Sessions: 1 Processed: 12 Uptime: 7m 55s
CPU: 0% Memory : 118M Last used: 1m 54s ago
* PID: 20784 Sessions: 1 Processed: 25 Uptime: 7m 53s
CPU: 0% Memory : 97M Last used: 1m 53s ago
* PID: 20833 Sessions: 1 Processed: 6 Uptime: 7m 51s
CPU: 0% Memory : 117M Last used: 1m 51s ago
* PID: 20889 Sessions: 1 Processed: 3 Uptime: 7m 50s
CPU: 0% Memory : 84M Last used: 5m 49s ago
* PID: 20951 Sessions: 1 Processed: 19 Uptime: 7m 48s
CPU: 0% Memory : 99M Last used: 1m 47s ago
* PID: 21016 Sessions: 1 Processed: 25 Uptime: 7m 46s
CPU: 0% Memory : 100M Last used: 1m 46s ago
* PID: 21068 Sessions: 1 Processed: 5 Uptime: 7m 44s
CPU: 0% Memory : 94M Last used: 1m 44s ago
* PID: 21128 Sessions: 1 Processed: 6 Uptime: 7m 42s
CPU: 0% Memory : 85M Last used: 1m 42s ago
* PID: 21180 Sessions: 1 Processed: 23 Uptime: 7m 41s
CPU: 0% Memory : 99M Last used: 1m 40s ago
* PID: 21238 Sessions: 1 Processed: 10 Uptime: 7m 39s
CPU: 0% Memory : 97M Last used: 1m 38s ago
* PID: 21290 Sessions: 1 Processed: 14 Uptime: 7m 37s
CPU: 0% Memory : 85M Last used: 1m 37s ago
* PID: 21341 Sessions: 1 Processed: 9 Uptime: 7m 35s
CPU: 0% Memory : 97M Last used: 1m 35s ago
* PID: 21399 Sessions: 1 Processed: 13 Uptime: 7m 33s
CPU: 0% Memory : 98M Last used: 1m 33s ago
* PID: 21448 Sessions: 1 Processed: 11 Uptime: 7m 32s
CPU: 0% Memory : 91M Last used: 5m 31s ago
* PID: 21500 Sessions: 1 Processed: 12 Uptime: 7m 30s
CPU: 0% Memory : 95M Last used: 1m 29s ago
* PID: 21557 Sessions: 1 Processed: 17 Uptime: 7m 28s
CPU: 0% Memory : 98M Last used: 1m 27s ago
* PID: 21606 Sessions: 1 Processed: 8 Uptime: 7m 26s
CPU: 0% Memory : 85M Last used: 1m 26s ago
* PID: 21657 Sessions: 1 Processed: 22 Uptime: 7m 24s
CPU: 0% Memory : 86M Last used: 1m 24s ago
* PID: 21715 Sessions: 1 Processed: 13 Uptime: 7m 22s
CPU: 0% Memory : 97M Last used: 1m 22s ago
* PID: 21844 Sessions: 1 Processed: 8 Uptime: 7m 21s
CPU: 0% Memory : 90M Last used: 5m 20s ago
* PID: 21912 Sessions: 1 Processed: 21 Uptime: 7m 19s
CPU: 0% Memory : 96M Last used: 1m 18s ago
* PID: 21969 Sessions: 1 Processed: 7 Uptime: 7m 17s
CPU: 0% Memory : 94M Last used: 1m 16s ago
* PID: 22052 Sessions: 1 Processed: 28 Uptime: 7m 15s
CPU: 0% Memory : 121M Last used: 1m 14s ago
* PID: 22119 Sessions: 1 Processed: 16 Uptime: 7m 13s
CPU: 0% Memory : 95M Last used: 1m 13s ago
* PID: 22190 Sessions: 1 Processed: 21 Uptime: 7m 11s
CPU: 0% Memory : 99M Last used: 1m 11s ago
* PID: 22262 Sessions: 1 Processed: 8 Uptime: 7m 10s
CPU: 0% Memory : 85M Last used: 1m 9s ago
* PID: 22321 Sessions: 1 Processed: 14 Uptime: 7m 8s
CPU: 0% Memory : 85M Last used: 1m 8s ago
* PID: 22373 Sessions: 1 Processed: 8 Uptime: 7m 6s
CPU: 0% Memory : 117M Last used: 1m 5s ago
* PID: 22429 Sessions: 1 Processed: 14 Uptime: 7m 4s
CPU: 0% Memory : 95M Last used: 1m 4s ago
* PID: 22488 Sessions: 1 Processed: 19 Uptime: 7m 2s
CPU: 0% Memory : 85M Last used: 1m 2s ago
* PID: 22542 Sessions: 1 Processed: 8 Uptime: 7m 0s
CPU: 0% Memory : 94M Last used: 1m 0s ago
* PID: 22601 Sessions: 1 Processed: 16 Uptime: 6m 59s
CPU: 0% Memory : 96M Last used: 58s ago
* PID: 22655 Sessions: 1 Processed: 6 Uptime: 6m 57s
CPU: 0% Memory : 96M Last used: 56s ago
* PID: 22714 Sessions: 1 Processed: 24 Uptime: 6m 55s
CPU: 0% Memory : 96M Last used: 55s ago
* PID: 22772 Sessions: 1 Processed: 8 Uptime: 6m 53s
CPU: 0% Memory : 97M Last used: 53s ago
* PID: 22822 Sessions: 1 Processed: 21 Uptime: 6m 51s
CPU: 0% Memory : 96M Last used: 51s ago
* PID: 22878 Sessions: 1 Processed: 13 Uptime: 6m 50s
CPU: 0% Memory : 115M Last used: 49s ago
* PID: 22938 Sessions: 1 Processed: 14 Uptime: 6m 48s
CPU: 0% Memory : 86M Last used: 47s ago
* PID: 23008 Sessions: 1 Processed: 22 Uptime: 6m 46s
CPU: 0% Memory : 96M Last used: 45s ago
* PID: 23060 Sessions: 1 Processed: 16 Uptime: 6m 44s
CPU: 0% Memory : 117M Last used: 43s ago
* PID: 23118 Sessions: 1 Processed: 6 Uptime: 6m 42s
CPU: 0% Memory : 94M Last used: 42s ago
* PID: 23171 Sessions: 1 Processed: 9 Uptime: 6m 40s
CPU: 0% Memory : 117M Last used: 40s ago
* PID: 23230 Sessions: 1 Processed: 11 Uptime: 6m 39s
CPU: 0% Memory : 118M Last used: 38s ago
* PID: 23284 Sessions: 1 Processed: 3 Uptime: 6m 37s
CPU: 0% Memory : 84M Last used: 36s ago
* PID: 23341 Sessions: 1 Processed: 8 Uptime: 6m 35s
CPU: 0% Memory : 94M Last used: 34s ago
* PID: 23399 Sessions: 1 Processed: 6 Uptime: 6m 33s
CPU: 0% Memory : 94M Last used: 33s ago
* PID: 23452 Sessions: 1 Processed: 4 Uptime: 6m 31s
CPU: 0% Memory : 94M Last used: 2m 31s ago
* PID: 23503 Sessions: 1 Processed: 12 Uptime: 6m 29s
CPU: 0% Memory : 95M Last used: 29s ago
* PID: 23562 Sessions: 1 Processed: 23 Uptime: 6m 28s
CPU: 0% Memory : 96M Last used: 27s ago
* PID: 23613 Sessions: 1 Processed: 14 Uptime: 6m 26s
CPU: 0% Memory : 85M Last used: 25s ago
* PID: 23667 Sessions: 1 Processed: 17 Uptime: 6m 24s
CPU: 0% Memory : 117M Last used: 23s ago
* PID: 23727 Sessions: 1 Processed: 20 Uptime: 6m 22s
CPU: 0% Memory : 96M Last used: 22s ago
* PID: 23857 Sessions: 1 Processed: 20 Uptime: 6m 20s
CPU: 0% Memory : 96M Last used: 20s ago
* PID: 23914 Sessions: 1 Processed: 14 Uptime: 6m 19s
CPU: 0% Memory : 85M Last used: 18s ago
* PID: 23977 Sessions: 1 Processed: 20 Uptime: 6m 17s
CPU: 0% Memory : 86M Last used: 16s ago
* PID: 24031 Sessions: 1 Processed: 8 Uptime: 6m 15s
CPU: 0% Memory : 117M Last used: 14s ago
* PID: 24091 Sessions: 1 Processed: 13 Uptime: 6m 13s
CPU: 0% Memory : 117M Last used: 13s ago
* PID: 24143 Sessions: 1 Processed: 12 Uptime: 6m 11s
CPU: 0% Memory : 95M Last used: 11s ago
* PID: 24212 Sessions: 1 Processed: 14 Uptime: 6m 10s
CPU: 0% Memory : 98M Last used: 9s ago
* PID: 24271 Sessions: 1 Processed: 25 Uptime: 6m 8s
CPU: 0% Memory : 96M Last used: 7s ago
* PID: 24328 Sessions: 1 Processed: 19 Uptime: 6m 6s
CPU: 0% Memory : 95M Last used: 5s ago
* PID: 24382 Sessions: 1 Processed: 9 Uptime: 6m 4s
CPU: 0% Memory : 95M Last used: 3s ago
* PID: 24440 Sessions: 1 Processed: 10 Uptime: 6m 2s
CPU: 0% Memory : 94M Last used: 2s ago
* PID: 24495 Sessions: 1 Processed: 7 Uptime: 6m 0s
CPU: 0% Memory : 94M Last used: 2m 0s ago
* PID: 24547 Sessions: 1 Processed: 8 Uptime: 5m 59s
CPU: 0% Memory : 85M Last used: 5m 58s ago
* PID: 24611 Sessions: 1 Processed: 13 Uptime: 5m 57s
CPU: 0% Memory : 95M Last used: 1m 56s ago
* PID: 24665 Sessions: 1 Processed: 4 Uptime: 5m 55s
CPU: 0% Memory : 117M Last used: 1m 54s ago
* PID: 24729 Sessions: 1 Processed: 6 Uptime: 5m 53s
CPU: 0% Memory : 93M Last used: 1m 53s ago
* PID: 24781 Sessions: 1 Processed: 3 Uptime: 5m 51s
CPU: 0% Memory : 84M Last used: 1m 51s ago
* PID: 24839 Sessions: 1 Processed: 25 Uptime: 5m 49s
CPU: 0% Memory : 86M Last used: 1m 49s ago
* PID: 24906 Sessions: 1 Processed: 1 Uptime: 5m 48s
CPU: 0% Memory : 84M Last used: 5m 47s ago
* PID: 24973 Sessions: 1 Processed: 9 Uptime: 5m 46s
CPU: 0% Memory : 94M Last used: 1m 45s ago
* PID: 25031 Sessions: 1 Processed: 8 Uptime: 5m 44s
CPU: 0% Memory : 85M Last used: 1m 44s ago
* PID: 25090 Sessions: 1 Processed: 3 Uptime: 5m 42s
CPU: 0% Memory : 95M Last used: 1m 42s ago
* PID: 25144 Sessions: 1 Processed: 8 Uptime: 5m 40s
CPU: 0% Memory : 117M Last used: 1m 40s ago
* PID: 25196 Sessions: 1 Processed: 22 Uptime: 5m 38s
CPU: 0% Memory : 96M Last used: 1m 38s ago
* PID: 25258 Sessions: 1 Processed: 6 Uptime: 5m 37s
CPU: 0% Memory : 94M Last used: 1m 36s ago
* PID: 25313 Sessions: 1 Processed: 11 Uptime: 5m 35s
CPU: 0% Memory : 94M Last used: 1m 34s ago
* PID: 25372 Sessions: 1 Processed: 20 Uptime: 5m 33s
CPU: 0% Memory : 96M Last used: 1m 33s ago
* PID: 25426 Sessions: 1 Processed: 13 Uptime: 5m 31s
CPU: 0% Memory : 95M Last used: 1m 31s ago
* PID: 25478 Sessions: 1 Processed: 3 Uptime: 5m 29s
CPU: 0% Memory : 93M Last used: 1m 29s ago
* PID: 25545 Sessions: 1 Processed: 11 Uptime: 5m 28s
CPU: 0% Memory : 94M Last used: 1m 27s ago
* PID: 25607 Sessions: 1 Processed: 3 Uptime: 5m 26s
CPU: 0% Memory : 93M Last used: 1m 25s ago
* PID: 25658 Sessions: 1 Processed: 10 Uptime: 5m 24s
CPU: 0% Memory : 97M Last used: 1m 23s ago
* PID: 25720 Sessions: 1 Processed: 14 Uptime: 5m 22s
CPU: 0% Memory : 118M Last used: 1m 22s ago
* PID: 25850 Sessions: 1 Processed: 7 Uptime: 5m 20s
CPU: 0% Memory : 123M Last used: 1m 20s ago
* PID: 25909 Sessions: 1 Processed: 5 Uptime: 5m 18s
CPU: 0% Memory : 84M Last used: 1m 18s ago
* PID: 25973 Sessions: 1 Processed: 17 Uptime: 5m 17s
CPU: 0% Memory : 98M Last used: 1m 16s ago
* PID: 26028 Sessions: 1 Processed: 17 Uptime: 5m 15s
CPU: 0% Memory : 95M Last used: 1m 14s ago
* PID: 26095 Sessions: 1 Processed: 2 Uptime: 5m 13s
CPU: 0% Memory : 95M Last used: 1m 13s ago
* PID: 26146 Sessions: 1 Processed: 9 Uptime: 5m 11s
CPU: 0% Memory : 117M Last used: 1m 11s ago
* PID: 26201 Sessions: 1 Processed: 6 Uptime: 5m 9s
CPU: 0% Memory : 93M Last used: 1m 9s ago
* PID: 26264 Sessions: 1 Processed: 18 Uptime: 5m 7s
CPU: 0% Memory : 95M Last used: 1m 7s ago
* PID: 26318 Sessions: 1 Processed: 8 Uptime: 5m 6s
CPU: 0% Memory : 117M Last used: 1m 5s ago
* PID: 26373 Sessions: 1 Processed: 14 Uptime: 5m 4s
CPU: 0% Memory : 85M Last used: 1m 3s ago
* PID: 26439 Sessions: 1 Processed: 13 Uptime: 5m 2s
CPU: 0% Memory : 97M Last used: 1m 2s ago
* PID: 26494 Sessions: 1 Processed: 16 Uptime: 5m 0s
CPU: 0% Memory : 97M Last used: 1m 0s ago
* PID: 26547 Sessions: 1 Processed: 9 Uptime: 4m 58s
CPU: 0% Memory : 117M Last used: 58s ago
* PID: 26610 Sessions: 1 Processed: 3 Uptime: 4m 56s
CPU: 0% Memory : 93M Last used: 56s ago
* PID: 26665 Sessions: 1 Processed: 4 Uptime: 4m 55s
CPU: 0% Memory : 84M Last used: 54s ago
* PID: 26721 Sessions: 1 Processed: 3 Uptime: 4m 53s
CPU: 0% Memory : 84M Last used: 52s ago
* PID: 26777 Sessions: 1 Processed: 8 Uptime: 4m 51s
CPU: 0% Memory : 94M Last used: 51s ago
* PID: 26834 Sessions: 1 Processed: 15 Uptime: 4m 49s
CPU: 0% Memory : 117M Last used: 49s ago
* PID: 26895 Sessions: 1 Processed: 5 Uptime: 4m 47s
CPU: 0% Memory : 84M Last used: 47s ago
* PID: 26963 Sessions: 1 Processed: 8 Uptime: 4m 45s
CPU: 0% Memory : 97M Last used: 45s ago
* PID: 27020 Sessions: 1 Processed: 12 Uptime: 4m 44s
CPU: 0% Memory : 94M Last used: 43s ago
* PID: 27081 Sessions: 1 Processed: 6 Uptime: 4m 42s
CPU: 0% Memory : 94M Last used: 41s ago
* PID: 27136 Sessions: 1 Processed: 6 Uptime: 4m 40s
CPU: 0% Memory : 84M Last used: 40s ago
* PID: 27189 Sessions: 1 Processed: 5 Uptime: 4m 38s
CPU: 0% Memory : 96M Last used: 38s ago
Any idea what could be causing this and what i could tweak to prevent it?
EDIT
passenger.conf:
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /usr/bin/passenger_free_ruby;
passenger_max_pool_size 160;
passenger_max_request_queue_size 250;