WCF dropping connections (random timeouts, socket aborts, endpoint not found) -Port sharing

Question

I'm doing some wcf tracing on a sever with 800 clients (verbose clients) and I am seeing a lot of SocketConnection aborted after Process action 'http://tempuri.org/IConnectionRegister/ValidateUriRoute'. This is during a ** ** activity

I removed (maxed out) all possible throttling (binding, service, port sharing (both on custom binding and SMSvcHost.exe). Still a lot of random timeouts and socket aborted.

Also the 0000 activity is absolutely full (less than every second) of

1. "The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'."

2. "The I/O operation has been aborted because of either a thread exit or an application request"

Any help would be very much appreciate in identifying who is causing the dropping of connections.

I can also upload a sample svc trace if anyone is interested.

Stack trace of Socket Connection Abort

System.ServiceModel.Channels.SocketConnection.Abort() System.ServiceModel.Channels.TracingConnection.Abort() System.ServiceModel.Channels.InitialServerConnectionReader.Abort() System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.OnValidate()

......

System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame() System.Runtime.AsyncResult.Complete() System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive() System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame() System.Runtime.AsyncResult.Complete() System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete()

<customBinding>
    <binding name="externalRoutingcustomBinding">
      <binaryMessageEncoding>

      </binaryMessageEncoding>

      <security
          authenticationMode="UserNameOverTransport"
          allowInsecureTransport="true"
          enableUnsecuredResponse="true"
          >
      </security><tcpTransport> 
        <connectionPoolSettings groupName = "default" maxOutboundConnectionsPerEndpoint = "1000"/> 
      </tcpTransport> 

      </binding>


  <service name="ZPRoutingWorkflow.ZpRoutingService">
<endpoint address="Transactions" binding="customBinding" bindingConfiguration="externalRoutingcustomBinding"
 name="Transactions" contract="TopUpInterface.ITransactionsService">
    <identity>
        <dns value="appserver.test.com" />
    </identity>
 </endpoint>

    <endpoint address="Retry" binding="customBinding" bindingConfiguration="externalRoutingcustomBinding" name="Retry" contract="IRetryInterface.IRetryService">
    <identity>
        <dns value="appserver.test.com" />
    </identity>
 </endpoint>

Config summary:

Multiple net.tcp services over same port hosted in iis. One of the services is a routing service implementing IRequestReplyRouter. Basically all messages go through the router and the router connects internally to the services and passes them their messages