Using http2_push with nginX in conjunction with HAProxy does not work

Question

I can not get HTTP/2 push to work when nginX is configured behind HaProxy. It does work when nginX is hit directly by the web browser however.

Did a lot of research already but did not find any hints. Hope anybody knows what I am doing wrong. See configuration and further observations below.

Configuration

The relevant HaProxy (version 1.8.7) configuration is given by:

    frontend appname
        bind *:443 ssl crt certificate.pem alpn h2,http/1.1
        mode tcp
        use_backend app-http2 if { ssl_fc_alpn -i h2 }
        default_backend app

    backend app-http2
        mode tcp
        server lamp2 127.0.0.1:8002 check send-proxy

And the relevant nginX (version 1.14.0) configuration is as follows:

    http {
        # This is the one I would like to use
        server {
            listen       8002 http2 proxy_protocol;

            server_name  _;
            root         /usr/share/nginx/html;

            location / {
                http2_push /image.jpg;
            }
        }

        # This one can be accessed directly; and *does* work
        server {
            listen       8004 http2 ssl;

            ssl_certificate certificate.pem;
            ssl_certificate_key private.key;

            server_name  _;
            root         /usr/share/nginx/html;

            location / {
                http2_push /image.jpg;
            }
        }
    }

Observations

• In the nginx logs I can verify that both ways of accessing the content do use HTTP2.
• When I use Chrome to access the page I can see that push is used only when directly visiting nginX

Update 9 may 2018 Still not solved. But people seem to agree that it is a bug. I opened an issue at their issue tracker: https://trac.nginx.org/nginx/ticket/1549#ticket

Update 26 april 2018

It appears the problem is bigger than just the http2 push. If I log the $scheme nginX variable it is always set to http. Both when accessing from http as from http2.

So that obviously seems like the problem. However I am not sure how I can fix this. Haproxy is working tcp mode; thus will likely not be doing anything wrong.

A related (but possibly outdated) Stack Overflow topic is nginx $scheme variable behind load balancer. But that answer does not help solving this problem!

Update 25 april 2018

Still not working. But a step closer. Ran nghttp2 on both, and results are found below.

Both seem to have the /image.jpg resource embedded. But the one that goes via haproxy has it's scheme set to http; and not to https. As one can see in this diff:

I assume due to this; Chrome will not use this pushed resource. I am however not sure what causes this!

Does anyone have a clue?

---

The complete output of both commands:

    nghttp -nv https://127.0.0.1:8004/

    [  0.001] Connected
    The negotiated protocol: h2
    [  0.003] send SETTINGS frame 
    (niv=2)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=201, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=101, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=7, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=3, weight=1, exclusive=0)
    [  0.003] send HEADERS frame 
    ; END_STREAM | END_HEADERS | PRIORITY
    (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
    ; Open new stream
    :method: GET
    :path: /
    :scheme: https
    :authority: 127.0.0.1:8004
    accept: */*
    accept-encoding: gzip, deflate
    user-agent: nghttp2/1.25.0
    [  0.003] recv SETTINGS frame 
    (niv=3)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
    [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
    [  0.003] recv WINDOW_UPDATE frame 
    (window_size_increment=2147418112)
    [  0.003] send SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.003] recv SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.003] recv (stream_id=13) :method: GET
    [  0.003] recv (stream_id=13) :path: /image.jpg
    [  0.003] recv (stream_id=13) :scheme: https
    [  0.003] recv (stream_id=13) :authority: 127.0.0.1:8004
    [  0.003] recv (stream_id=13) accept-encoding: gzip, deflate
    [  0.003] recv (stream_id=13) user-agent: nghttp2/1.25.0
    [  0.003] recv PUSH_PROMISE frame 
    ; END_HEADERS
    (padlen=0, promised_stream_id=2)
    [  0.003] recv (stream_id=13) :status: 200
    [  0.003] recv (stream_id=13) server: nginx/1.14.0
    [  0.003] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:26 GMT
    [  0.003] recv (stream_id=13) content-type: text/html
    [  0.003] recv (stream_id=13) content-length: 638
    [  0.003] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
    [  0.003] recv (stream_id=13) etag: "5ae069c2-27e"
    [  0.003] recv (stream_id=13) accept-ranges: bytes
    [  0.003] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First response header
    [  0.004] recv DATA frame 
    ; END_STREAM
    [  0.004] recv (stream_id=2) :status: 200
    [  0.004] recv (stream_id=2) server: nginx/1.14.0
    [  0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:26 GMT
    [  0.004] recv (stream_id=2) content-type: image/jpeg
    [  0.004] recv (stream_id=2) content-length: 182884
    [  0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
    [  0.004] recv (stream_id=2) etag: "57656be2-2ca64"
    [  0.004] recv (stream_id=2) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First push response header
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=33248)
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32925)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.134] recv DATA frame 
    [  0.134] recv DATA frame 
    [  0.134] recv DATA frame 
    ; END_STREAM
    [  0.134] send GOAWAY frame 
    (last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])

and

    nghttp -nv https://127.0.0.1:8002/

    [  0.001] Connected
    The negotiated protocol: h2
    [  0.003] send SETTINGS frame 
    (niv=2)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=201, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=101, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=7, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=3, weight=1, exclusive=0)
    [  0.003] send HEADERS frame 
    ; END_STREAM | END_HEADERS | PRIORITY
    (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
    ; Open new stream
    :method: GET
    :path: /
    :scheme: https
    :authority: 127.0.0.1:8002
    accept: */*
    accept-encoding: gzip, deflate
    user-agent: nghttp2/1.25.0
    [  0.003] recv SETTINGS frame 
    (niv=3)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
    [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
    [  0.003] recv WINDOW_UPDATE frame 
    (window_size_increment=2147418112)
    [  0.003] send SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.004] recv SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.004] recv (stream_id=13) :method: GET
    [  0.004] recv (stream_id=13) :path: /image.jpg
    [  0.004] recv (stream_id=13) :scheme: http
    [  0.004] recv (stream_id=13) :authority: 127.0.0.1:8002
    [  0.004] recv (stream_id=13) accept-encoding: gzip, deflate
    [  0.004] recv (stream_id=13) user-agent: nghttp2/1.25.0
    [  0.004] recv PUSH_PROMISE frame 
    ; END_HEADERS
    (padlen=0, promised_stream_id=2)
    [  0.004] recv (stream_id=13) :status: 200
    [  0.004] recv (stream_id=13) server: nginx/1.14.0
    [  0.004] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:45 GMT
    [  0.004] recv (stream_id=13) content-type: text/html
    [  0.004] recv (stream_id=13) content-length: 638
    [  0.004] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
    [  0.004] recv (stream_id=13) etag: "5ae069c2-27e"
    [  0.004] recv (stream_id=13) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First response header
    [  0.004] recv DATA frame 
    ; END_STREAM
    [  0.004] recv (stream_id=2) :status: 200
    [  0.004] recv (stream_id=2) server: nginx/1.14.0
    [  0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:45 GMT
    [  0.004] recv (stream_id=2) content-type: image/jpeg
    [  0.004] recv (stream_id=2) content-length: 182884
    [  0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
    [  0.004] recv (stream_id=2) etag: "57656be2-2ca64"
    [  0.004] recv (stream_id=2) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First push response header
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=33406)
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.044] recv DATA frame 
    [  0.044] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.044] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.044] recv DATA frame 
    [  0.044] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.045] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    ; END_STREAM
    [  0.046] send GOAWAY frame 
    (last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])

Answer 1

I ended up opening a ticket on the NginX tracker. And it got fixed. The fix is available in the recently released 1.15.1 version.

Bugfix: HTTP/2 server push did not work if SSL was terminated by a proxy server in front of nginx.

Thanks for the help!

Answer 2

It appears that Chrome makes sure that if a push is sent on a stream that corresponds to a request served over the https scheme, the promise also uses the same scheme: https://chromium.googlesource.com/chromium/src/+/master/net/spdy/chromium/spdy_session.cc#1766

I'm not sure that the check is correct: the RFC 7540 says that the server sending a PUSH_PROMISE should be authoritative. For http that means that the host name matches, so the browser should be able to process the file.

That said, even if the browser accepted the push, it would only use it if a request against http://127.0.0.1:8002/image.jpg were issued by the browser. If the HTML was obtained via https and it requested /image.jpg, then i'm not sure if the browser would accept to fetch http://127.0.0.1:8002/image.jpg.

Which brings us to nginx setting the scheme as http. I presume this is due to the fact that haproxy does the SSL termination, so ngnix sees a cleartext connection incoming, and as far as it's concerned, the scheme is http. I don't know enough about ngnix to suggest a fix for this.