Random HTTP 408 on POST requests and no body is transferred

Question

Some time ago we have noticed a number of HTTP 408 in the nginx connections log. All such requests are POST requests with a request time equal to nginx client_body_timeout. We have gathered a tcp dump and decrypt tls traffic. It looks like the client starts a request, sends headers with specied non-zero content-length header but then after a delimeter between headers and a body there is no data transmitted at all from the client side. Nginx politely waits for any data and after a period equal to client_body_timeout gently closes the connection. Could anyone clarify what could be the cause of such behavior? Some additional information:

• nginx version is 1.20.2 and the problem persists for a year or more.
• The problem is sporadical. There is no dependence with daily load peaks but it looks like that some client's ip have such error more frequently that others.
• Daily average number of HTTP 408 is ~1k per 110M of total POST requests.
• The problem persists on different hosts, with different nginx instances and different services.
• We tried to forward traffic through amazon alb but the problem persists.
• The requests which lead to HTTP 408 are sent via firefox, chrome and our ios and android applications.
• We cant reproduce the problem via tests even generating a large number of identical to client requests sent via jmeter.
• The problem has very little impact on client's experience as we have no complains about these errors.

Answer 1

We are having the same problem. In our case nginx server returns HTTP 408 code when receiving a HTTPS POST request with a large header (20 Kb) but 0 length body - the same request with smaller header works correctly. Adding just a 1 char body(' ') solved the problem in our case but I don't really understand the reason.