benchmark HTTP1.1/HTTP2 in python

Question

I did some benchmark on http1.1/http2 with python, the code is as simple as repeating a Google search request many times. The result is interesting: the http2 version is slower considerably. (i tried both pycurl/httpx libraries) Can someone explain why this happens?

Update: this is the httpx version code: (first pip install httpx[http2])

import time
import httpx
client = httpx.Client(http2=True)
start = time.time()
for _ in range(100):
    response = client.get("https://www.google.com/search?q=good")
    response.raise_for_status()
print(time.time() - start)

Answer 1

So it's important to understand what HTTP/2 aims to solve and what it doesn't.

HTTP/2 aims to be more efficient for multiple requests to the same site by adding multiplexing to the HTTP protocol. Basically HTTP/1.1 blocks the whole connection while a request is in process, but HTTP/2 doesn't — so allows other requests to be made during this time.

What this means, is that a single request (like you are doing) is no better under HTTP/2 than it is under HTTP/1.1. In fact it may even be slightly slower due to some extra set-up messages sent at the beginning of each HTTP/2 connection, which aren't needed under HTTP/1.1. Though I'm surprised if this difference was noticeable to be honest, so can you give more details of how much of a slowness there was? It may also point to less efficient code in the HTTP/2 implementation you are using. Can you share the code?

Even in the browser context, if you look at a highly optimised site like Google's home page (probably the most visited page on the internet, and run by a company that knows a LOT about the web and how to optimise web pages), then you may also not see a difference. The Google home page basically renders in a single request as all critical requests are inlined to make it as fast as possible — no matter whether using HTTP/1.1 or HTTP/2.

However a typical page loaded in the browser involving tens or even hundreds of requests the advantage of HTTP/2 is often very noticable.

And if you take an extreme site with lots of small requests (which is what HTTP/2 really excels at!), then the difference is really noticeable.

***** Edit, looking at the good you provided ****

Regarding your particular test case I was able to repeat this for Google, but not for other sites.

The difference seems too great to be due to HTTP/1.1 and HTTP/2 so I suspected that HTTP/1.1 was reusing the connection, but HTTP/2 was not. Moving the connection setup into the for loop gave the same slow results for both, and similar to previous HTTP/2 timings seemingly confirming this.

import time
import httpx
start = time.time()
for _ in range(100): 
    client = httpx.Client(http2=True)
    response = client.get("https://www.google.com/search?q=good")
    response.raise_for_status()
print(time.time() - start)

Similarly changing keepalives to 0 also slowed HTTP/1.1 down to match HTTP/2:

client = httpx.Client(http2=False,limits=httpx.Limits(max_keepalive_connections=0))

Keepalive is no longer a concept in HTTP/2 (connections are kept alive by default until the client deems it no longer necessary to keep around).

So this seems to be a problem with httpx's HTTP/1 handling (they do note it is experimental), rather than a problem with the protocol itself.

Finally moving to this code style, seemed to bring HTTP/2 stats back into line with HTTP/1.1:

import time
import httpx

with httpx.Client(http2=False) as client:
    start = time.time()
    for _ in range(100):
        response = client.get("https://www.google.com/search?q=good")
        response.raise_for_status()
    print(time.time() - start)

But at this point. Google was getting bored of me spamming their servers and returning a 429 Client Error: Too Many Requests... error.

When I tried to repeat the same issues on my own server, and then on stackoverflow.com, I couldn't - HTTP/1.1 and HTTP/2 were similar speeds. Not sure if Google was doing some caching on their side if same connection which helped speed it up.

Anyway, point is, this seems to be an implementation specific issue and not something to do with the HTTP protocol itself.

Answer 2

https://github.com/dalf/pyhttp-benchmark might help in a way or another.

See:

• results for httpx 0.18.1
• results for httpx 0.16.1
• Labels:
  • httpx_uvloop_True = httpx, using uvloop and http2
  • httpx_uvloop_False = httpx, using uvloop and http1 only

TLDR: with httpx, when using http2

• avoid large content (>64kb).
• avoid sequential requests
• prefer parallel requests