Ruby SSL TCP Server stuck when more than 250 concurrent connexions

Question

I am developing a SSL TCP Server in ruby and testing it against a multi threads client. When the number of threads on the client side is less than 190, there is no problem on the server, all the messages are received correctly. But once I increase the number of threads on the client side over 195, two problems pop up :

Problem 1 : Exception ECONNABORTED on the server side

/usr/local/rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/openssl/ssl.rb:232:in `accept': Software caused connection abort - accept(2) (Errno::ECONNABORTED)
        from /usr/local/rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/openssl/ssl.rb:232:in `accept'
        from server.rb:30:in `block (2 levels) in start_server'

I am able to workaround this by restarting the accept loop in the exception handler.

Problem 2 : Server stuck When I increase the number of thread on the client side (e.g. 250), after several seconds, the server is frozen, i.e. no exception and no new connection allowed. This one is really annoying because there is no way on the server side to know it is frozen.

OS : FreeBSD 10.1 Ruby version : 2.2.1 (tried 2.1.5 as well)

---

Server code :

loop do
    server = TCPServer.new(ip_address, port)
    sslContext = OpenSSL::SSL::SSLContext.new
    sslContext.cert = OpenSSL::X509::Certificate.new(File.open("cert/cert.pem"))
    sslContext.key = OpenSSL::PKey::RSA.new(File.open("cert/key.pem"), SSL_PASSWORD)
    sslServer = OpenSSL::SSL::SSLServer.new(server, sslContext)

    loop do
        Thread.new(sslServer.accept) do |connection|
          begin
            messageIn = connection.gets
            connection.close
          rescue Exception => ex
            puts "Exception in main loop : " + ex.message
            puts "Backtrace : " + ex.backtrace.join("\n")
          end
        end
    end
  end
end

Client code :

def create_client(host, port)
  begin
    socket = TCPSocket.open(host,port)
    ssl_context = OpenSSL::SSL::SSLContext.new()
    ssl_context.cert = OpenSSL::X509::Certificate.new(File.open("lib/cert/cert.pem"))
    ssl_context.key = OpenSSL::PKey::RSA.new(File.open("lib/cert/key.pem"), SSL_PASSWORD)
    ssl_context.ssl_version = :SSLv3
    ssl_context.ssl_timeout = 10
    ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
    ssl_socket.sync_close = true
    ssl_socket.connect
  rescue Exception => ex
    puts "Exception in create_client"
    sleep 1
    return create_client(host, port )
  end

  return ssl_socket
end

........

for j in 1..10 do
      threads = []
      for i in 1..n.to_i do
          threads << Thread.new do
            begin
              socket = create_client(ip, port)
              socket.puts("hello")
              socket.flush
              socket.close
            rescue Exception => ex
              puts "Exception"
            end
          end
      end

      threads.each(&:join)
    end

Answer 1

Sounds like you're running into networking limits on the server side. You probably need to increase the queue length with sysctl kern.ipc.soacceptqueue=250 (or set it permanently in /etc/sysctl.conf). You can check the application connection queue with netstat -Lan.

Answer 2

I finally found a solution (maybe not the only one) with the following content of /etc/sysctl.conf which I found here https://calomel.org/freebsd_network_tuning.html

kern.ipc.maxsockbuf=4194304  # (default 2097152)
net.inet.tcp.sendbuf_max=4194304  # (default 2097152)
net.inet.tcp.recvbuf_max=4194304  # (default 2097152)
net.inet.tcp.cc.algorithm=htcp  # (default newreno)
net.inet.tcp.cc.htcp.adaptive_backoff=1 # (default 0 ; disabled)
net.inet.tcp.cc.htcp.rtt_scaling=1 # (default 0 ; disabled)
net.inet.ip.forwarding=1      # (default 0)
net.inet.ip.fastforwarding=1  # (default 0)
kern.ipc.soacceptqueue=1024  # (default 128 ; same as kern.ipc.somaxconn)
net.inet.tcp.mssdflt=1460  # (default 536)
net.inet.tcp.minmss=1300   # (default 216)
net.inet.tcp.rfc1323=1  # (default 1)
net.inet.tcp.rfc3390=1  # (default 1)
net.inet.tcp.sack.enable=1  # (default 1)
net.inet.tcp.tso=0   # (default 1)
net.inet.tcp.nolocaltimewait=1  # (default 0)
net.inet.tcp.experimental.initcwnd10=1        # (default 1 for FreeBSD 10.1)
net.inet.tcp.syncache.rexmtlimit=0  # (default 3)
net.inet.ip.rtexpire=2       # (default 3600)
net.inet.ip.rtminexpire=2    # (default 10  )
net.inet.tcp.syncookies=0  # (default 1)
net.inet.ip.check_interface=1         # verify packet arrives on correct interface (default 0)
net.inet.ip.process_options=0         # ignore IP options in the incoming packets (default 1)
net.inet.ip.redirect=0                # do not send IP redirects (default 1)
net.inet.ip.stealth=1                 # do not reduce the TTL by one(1) when a packets goes through the firewall (default 0)
net.inet.icmp.drop_redirect=1         # no redirected ICMP packets (default 0)
net.inet.tcp.drop_synfin=1            # SYN/FIN packets get dropped on initial connection (default 0)
net.inet.tcp.fast_finwait2_recycle=1  # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0)
net.inet.tcp.icmp_may_rst=0           # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.msl=5000                 # 5s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0     # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
net.inet.udp.blackhole=1              # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2              # drop tcp packets destined for closed ports (default 0)
security.bsd.see_other_uids=0         # users only see their own processes. root can see all (default 1)

I am now able to run 1000 threads on the client side without any exception or freeze on the server side.