I've been experiencing a very interesting problem in openresty, which is running under heavy load.
Some of my users temporarily get a 500 error on their requests, but this error is logged in the "access.log" instead of the "error.log". This is so rare that it is very difficult, if not impossible, for me to detect. This is a fact, although I was quite surprised when I first saw it. It happens in about 5-10 out of every 10,000 requests and it doesn't happen every time. Also, when I make the same request myself, I get 200 responses, as I should.
Do you have any idea about the possible reasons for this?
My guess is that it's due to the filtering and storing thousands of data in memory with lua_shared_dict.
Is there any possibility that comes to your mind?