Apache proxy is not doing connection re-use with either prefork or worker MPM

Question

Background Info

We have a Dell R620 telecom based enterprise application server runs on RHEL 5 platform & deployed with both Apache web server & OCCAS application server. We have proprietary phones which connects to the application server as clients & hits the Apache for all HTTP based requests. The Apache server is configured to proxy/forward the requests to OCCAS.

The Problem

The proprietary phones will send around 350 HTTP requests (during start-up) within 35 seconds i.e., around 10 requests per second. We could clearly see the connection between the phone & Apache is reusing the connections, whereas, the connections between Apache i.e., mod_proxy & OCCAS is NOT reusing connection, ending up creating one connection per request. The connection between Apache & OCCAS is closed by Apache/proxy (as we see FIN from proxy) right after the request is serviced. So, we could see 350 sockets in TIME_WAIT state for a minute after all 350 requests are serviced. Our system is supposed to handle 500 such phones. Now, if you can imagine when all 500 phones making 350 requests pretty much at the same time, it would require 175,000 (350 * 500) ephemeral ports! Hence, I was trying to make connection reuse to work between Apache proxy & OCCAS, for which I'm facing issues.

I have posted below the configurations we have & different things we tried & failed to make connection reuse to work.

Our configurations

1) The prefork & worker MPM configured in our system in /etc/httpd/conf/httpd.conf is as follows & we are using the default prefork MPM.

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

<IfModule worker.c>
StartServers         8
MaxClients         150
MinSpareThreads      5
MaxSpareThreads     20
ThreadsPerChild    256
MaxRequestsPerChild  4000
</IfModule>

2) Keepalive related parameters in /etc/httpd/conf/httpd.conf is as follows:

[root@host ~]# grep -i keepalive /etc/httpd/conf/httpd.conf | grep -v "#"
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
[root@host ~]#

3) The proxy configuration is as follows:

<VirtualHost *:80>

<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>

<Location /ds-webservice>
ProxyPass         http://{IP address of local host}:8001/ds-webservice retry=0
ProxyPassReverse  http://{IP address of local host}:8001/ds-webservice
</Location>
.
.
# Few more similar ProxyPass
.
.
</VirtualHost>

4) We use default prefork Apache:

[root@host ~]# httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

Tests tried but failed to achieve connection reuse

Test 1) We noticed the mod_proxy is sending 'Connection: Keep-alive' header (but NOT 'Keep-alive' header itself) in the forwarded requests whereas OCCAS does NOT respond with 'Connection' or 'Keep-alive' headers. Our initial suspect was Apache does not like the response in absence of these headers. So, as an experiment, we forwarded the requests to another Apache in which we did get back 'Connection' or 'Keep-alive' headers in response from other Apache to mod_proxy, and still the mod_proxy did NOT do connection reuse. So, we confirmed its the mod_proxy which has the issue (or may be some misconfiguration from our side).

Test 2) Researching a bit, we found a bug report on connection reuse with prefork in 2.2.0 - https://issues.apache.org/bugzilla/show_bug.cgi?id=38602. The bug 38602 is fixed in 2.2.1. However, based on the post Can I use Apache mod_proxy as a connection pool, under the Prefork MPM? we found the issue seem to persist even after upgrading. So, just to be sure, we upgraded our Apache from 2.2.3 to 2.2.26 but mod_proxy did NOT do connection reuse.

Test 3) We tried few tests with with different ProxyPass parameters (without changing anything in httpd.conf) & none of them helped i.e., mod_proxy did NOT do connection reuse: NOTE: Excluding (7), we added the below parameters next to 'ProxyPass http://{IP address of local host}:8001/ds-webservice' for our tests.
1. retry=0 (Our default configuration)
2. disablereuse=OFF
3. max=256 retry=0
4. proxyto=80
5. retry=0 keepalive=ON
6. retry=300 smax=5 max=20 ttl=120
7. And lastly, we added keep-alive specific parameters inside the proxy configuration hoping it'll override the keep-alive parameters in http.conf :

<VirtualHost *:80>

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 100
ProxyRequests Off

<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>

<Location /ds-webservice>
ProxyPass         http://{IP address of local host}:8001/ds-webservice retry=0
ProxyPassReverse  http://{IP address of local host}:8001/ds-webservice
</Location>
.
.
# Few more similar ProxyPass
.
.
</VirtualHost>

Test 4) Based on the post Can I use Apache mod_proxy as a connection pool, under the Prefork MPM?, we switched to prefork Apache to worker Apache by using httpd.worker on 2.2.26 & mod_proxy did NOT do connection reuse.

It seems we are missing something very fundamental here. Can you guys point me if there is anything we did wrong either in ProxyPass or prefork configuration as both are quite related? Any pointers will be of great help.

Thanking you in advance!

Regards
Goutham Prasad

Answer 1

Chances are good that either your clients are making requests under HTTP protocol 1.0 (where keepalives need to be explicitly enabled via headers) or that either request or response is requesting connections be closed with the Connection: close header. See http://en.wikipedia.org/wiki/HTTP_persistent_connection

The best way to determine what is going on would be to post headers from packet traces of:

1. The request from the client to the proxy
2. The request from the proxy to OCCAS
3. The response from OCCAS to the proxy

This would enable seeing what protocol version and headers are in play.