Haskell simpleHTTP of Network.HTTP.Conduit performing slowly for get requests

Question

In my haskell code I've imported Network.HTTP.Conduit as

import Network.HTTP.Conduit

In my main function I make a simple GET request using simpleHTTP

main = do
     response <- simpleHttp "https://github.com/trending?l=ruby"
     return ()

It took 6 minutes 42 secs to complete 100 api requests

time for NUM in `seq 1 1 100`; do ./Testhttp; done
real    6m42.839s
user    0m12.115s
sys 0m2.652s

whereas the ruby alternative just takes 153 secs for 100 api calls using Net::HTTP.get(URI.parse("https://github.com/trending?l=ruby"))

Am I doing something wrong in my haskell code? What are the performant and efficient alternatives to simpleHTTP?

Answer 1

The documentation for simpleHttp says:

Note: This function creates a new Manager. It should be avoided in production code.

Your code creates a new manager for each request. If you change it to reuse a single manager, it could be a lot faster. You can use newManager to create a manager. For example:

import Network.HTTP.Conduit
main = do
    request <- parseUrl "https://github.com/trending?l=ruby"
    manager <- newManager conduitManagerSettings
    _response <- httpLbs request manager
    return ()

Answer 2

My guess would be that the Haskell library you are using is doing something like an IPv6 DNS request that times out before falling back to IPv4, and Go and Ruby are doing IPv4 requests directly. A few seconds per request is a likely DNS timeout duration and doesn't have any other possible explanation that I can see.