I have a strange situation. I have a web page that makes several AJAX calls to a php file. Some of these calls get stuck for several seconds to minutes - noticed that in Inspect Element on Google Chrome -> Network.
I checked the apache log, and noticed that these calls take long to process as can be seen in the time taken to serve the php page. The page has some db calls, but the sqls are super fast and don't hold up anything. In fact, I commented out the code in php page but still the apache log shows these calls take long to load.
I checked server-status and noticed the processes stuck in W status for several minutes. Since the ajax calls happen in loop, we have several calls going at the same time to the same page, with different GET variables. Some of them finish quite quickly and others take time. It is not the same calls that get delayed every time.. the calls that go into W status change every time (GET variables are different).
The same code works very well in our test server. The problem is in production server. Production has SSL enabled, but otherwise I don't see any difference between the servers.
I tried pstack PID on the apache processes that are getting stuck, but I am not sure how to read the resulting output. Here it is.
root# pstack 14759
#0 0x00007f0352a4d1d7 in flock () from /lib64/libc.so.6
#1 0x00007f034b50742a in ?? () from /etc/httpd/modules/libphp5.so
#2 0x00007f034b50752e in ?? () from /etc/httpd/modules/libphp5.so
#3 0x00007f034b5062e7 in php_session_start () from /etc/httpd/modules/libphp5.so
#4 0x00007f034b506aa9 in ?? () from /etc/httpd/modules/libphp5.so
#5 0x00007f034b685568 in ?? () from /etc/httpd/modules/libphp5.so
#6 0x00007f034b61f42b in execute () from /etc/httpd/modules/libphp5.so
#7 0x00007f034b5fe026 in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#8 0x00007f034b5abfb6 in php_execute_script () from /etc/httpd/modules/libphp5.so
#9 0x00007f034b68d132 in ?? () from /etc/httpd/modules/libphp5.so
#10 0x000055d5e59ea840 in ap_run_handler ()
#11 0x000055d5e59eac09 in ap_invoke_handler ()
#12 0x000055d5e59f7da0 in ap_process_request ()
#13 0x000055d5e59f4cc8 in ?? ()
#14 0x000055d5e59f0f20 in ap_run_process_connection ()
#15 0x000055d5e59fc9fb in ?? ()
#16 0x000055d5e59fce72 in ?? ()
#17 0x000055d5e59fddc7 in ap_mpm_run ()
#18 0x000055d5e59d487e in main ()```
Can someone help me figure out why this process is waiting?
Btw, I have disabled mod_evasive so it cannot be due to some kind of rate limiting. But it seems like the issue is very similar to that I would observe when I hit a rate limit.
php_session_start ()in the pstack was the hint. Searched for sessions in ajax/php calls. And here was the problem. Ajax does async calls and php cannot handle multiple calls at the same time. Each call locks the file for the session and then the session has to become free before the next call can lock it.I changed my sessions from files to mysql db and this solved the problem.