Dropped connections to server. Haproxy + Nginx + Unicorn

Question

Some requests to my site are failed with "Server unexpectedly dropped the connection" and there is no errors in error logs.

RPM is 1-1.2k.

Do you have any ideas?

Here is configs of haproxy, nginx and unicorn:

https://gist.github.com/releu/59b7f1d649c681856320

Answer 1

Before you share logs which is critical you have misconfiguration:

1. Nginx worker_connections 3000 Reason: keep it 2-3x more than actual active connections number

2. Nginx worker_rlimit_nofile 20000. Reason: each single connection requires 2 file descriptors for serving request via proxying

3. Nginx absence of location that serves static files straight from file system bypass backend. Reason: if you pass any static file via backend it reach the limit very quick

4. Unicorn worker_processes 4. Reason: 4 workers are not enough to serve 1K RPM

5. System: I guess there is lack of kernel optimisation. Take a look Unicorn System Kernel tips

Answer 2

This looks like it could be an issue with your Unicorn config - it's set to have a backlog of 1024, which could cause some issues despite being the default setting.

From the Unicorn documentation: "If you are running unicorn on multiple machines, lowering this number can help your load balancer detect when a machine is overloaded and give requests to a different machine."

It may also be worth monitoring your system resources when this happens. Please let us know how this goes for you!