spdy3's name/value block can't be decompressed SOMETIMES with python

Question

I was building a spdy proxy server for chrome.

When proxy requests was sent from chrome, usually I would receive 3 frames:

1. a SETTINGS frame ( parsed correctly)
2. a SYN_STREAM frame of https' CONNECT request ( parsed correctly, too )
3. another SYN_STREAM frame which name/value block could not be decompressed (Error -3 while decompressing data: incorrect header check)

The last frame was quite strange, because I could parse the 1st SYN_STREAM which also had name/value block.

I'd been trying to solve the problem for 2 full days, and even switched the spdy version from 2 to 3, but still couldn't work it out.

Here is my decompresion function(Python 3.3):

def decompress(buf):
    decompressor = zlib.decompressobj(zdict=DICT3)
    # DICT3 is the dictionary of spdy3 to decompress NV block
    return decompressor.decompress(buf)

Here is a full Example of the logs:

(1)2014-01-06 14:44:55,070 proxy           DEBUG: [49036176] on_read b'\x80\x03\x00\x04\x00\x00\x00\x14\x00\x00\x00\x02\x00\x00\x00\x04\x00\x00\x03\xe8\x00\x00\x00\x07\x00\xa0\x00\x00\x80\x03\x00\x01\x00\x00\x00\xbf\x00\x00\x00\x01\x00\x00\x00\x00\x00\x008\xea\xe3\xc6\xa7\xc2\x02e%\x10\xb6\x82\x16\x08\\\xc0\xd4U\x90X\xa9\x97\x0c.&\xd8\xad\xe0\x89\x99\xdd\xd9\xdf\xcf\xcf\xd59\x04\xac\xb8 \x11\\l\xf1\x01\x04PbNfAb\xa5^r~\xae\x95\x89\x891@\x00\x81\x12\xb3\x15\xf6\x84\x8d\x9a\xa1r|\xf3\xab2sr\x12\xf5M\xf5\x0c\x144\xc23\xf3R\xf2\xcb\x8b\x15\xfcB\x14\xcc\xf4\x0c\xad\x15\xc2\xfd\xc3\xcdL4\x811\x06\x8c\xdf\xf0\xd4$\xef\xcc\x12}Scs=c3\x05\ro\x8f\x10_\x1f\x1d\x85\x9c\xcc\xecT\x05\xf7\xd4\xe4\xec|M\x05\xe7\x0c`\x89\x96\xaaol\xa8g\xa0ghfj\xa0g\x06L\x0b\x89i\x89E\x99P]\x00\x00\x00\x00\xff\xff\x80\x03\x00\x01\x00\x00\x00)\x00\x00\x00\x03\x00\x00\x00\x00`\x00\xa2\xcc\x87\x00\x01\x94\x98\x93Y\x90X\xa9\x97\x9c\x9fkebb\x0c\x10@\x83\xd3\x87\x00\x00\x00\x00\xff\xff'
(2)2014-01-06 14:44:55,071 proxy           DEBUG: [49036176] >-- <SETTINGS SETTINGS_MAX_CONCURRENT_STREAMS=1000 SETTINGS_INITIAL_WINDOW_SIZE=10485760  [20]>
(3)2014-01-06 14:44:55,073 proxy           DEBUG: [49036176] >-- <SYN_STREAM #1->#0 [191]:host: alipay.com  :method: CONNECT    :version: HTTP/1.1  :path: alipay.com:443   user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36    >
(4)2014-01-06 14:44:55,074 proxy           DEBUG: [49036176] error Error -3 while decompressing data: incorrect header check
(5)2014-01-06 14:44:55,074 proxy           DEBUG: [49036176] zlib.e b'\x80\x03\x00\x01\x00\x00\x00)\x00\x00\x00\x03\x00\x00\x00\x00`\x00\xa2\xcc\x87\x00\x01\x94\x98\x93Y\x90X\xa9\x97\x9c\x9fkebb\x0c\x10@\x83\xd3\x87\x00\x00\x00\x00\xff\xff'
(6)2014-01-06 14:45:25,070 proxy           DEBUG: [49036176] on_read b'\x80\x03\x00\x03\x00\x00\x00\x08\x00\x00\x00\x01\x00\x00\x00\x05\x80\x03\x00\x01\x00\x00\x00)\x00\x00\x00\x05\x00\x00\x00\x00`\x00\xa2\xcc\x87\x00\x01\x94\x98\x93Y\x90X\xa9\x97\x9c\x9fkebb\x0c\x10@\x83\xd3\x87\x00\x00\x00\x00\xff\xff\x80\x03\x00\x03\x00\x00\x00\x08\x00\x00\x00\x03\x00\x00\x00\x05'
(7)2014-01-06 14:45:25,072 proxy           DEBUG: [49036176] >-- <RST_STREAM #1 [8]CANCEL>

Let me explain a little:

(1) Proxy received the raw bytes from socket, which containes 3 spdy frames

(2) A SETTINGS frame was parsed from the bytes of (1)

b'\x80\x03\x00\x04\x00\x00\x00\x14\x00\x00\x00\x02\x00\x00\x00\x04\x00\x00\x03\xe8\x00\x00\x00\x07\x00\xa0\x00\x00

(3) a SYN_STREAM frame was also parsed

b'\x80\x03\x00\x01\x00\x00\x00\xbf\x00\x00\x00\x01\x00\x00\x00\x00\x00\x008\xea\xe3\xc6\xa7\xc2\x02e%\x10\xb6\x82\x16\x08\\\xc0\xd4U\x90X\xa9\x97\x0c.&\xd8\xad\xe0\x89\x99\xdd\xd9\xdf\xcf\xcf\xd59\x04\xac\xb8 \x11\\l\xf1\x01\x04PbNfAb\xa5^r~\xae\x95\x89\x891@\x00\x81\x12\xb3\x15\xf6\x84\x8d\x9a\xa1r|\xf3\xab2sr\x12\xf5M\xf5\x0c\x144\xc23\xf3R\xf2\xcb\x8b\x15\xfcB\x14\xcc\xf4\x0c\xad\x15\xc2\xfd\xc3\xcdL4\x811\x06\x8c\xdf\xf0\xd4$\xef\xcc\x12}Scs=c3\x05\ro\x8f\x10_\x1f\x1d\x85\x9c\xcc\xecT\x05\xf7\xd4\xe4\xec|M\x05\xe7\x0c`\x89\x96\xaaol\xa8g\xa0ghfj\xa0g\x06L\x0b\x89i\x89E\x99P]\x00\x00\x00\x00\xff\xff'

(4) When the parsing continued, it encountered another SYN_STREAM

b'\x80\x03\x00\x01\x00\x00\x00)\x00\x00\x00\x03\x00\x00\x00\x00`\x00\xa2\xcc\x87\x00\x01\x94\x98\x93Y\x90X\xa9\x97\x9c\x9fkebb\x0c\x10@\x83\xd3\x87\x00\x00\x00\x00\xff\xff'

but somehow the name/value block part can't be decompressed

The 6-8 bytes tells the length of the remaining data (length:'\x00\x00)'=41），and the tail is '\x00\x00\xff\xff', so I should have splitted the frame correctly.

After the 'length bytes', there are 10 other bytes before the compressed name/value block.

(5) An error was captured, the trouble frame's bytes was listed, the actual compressed name/value block's bytes were:

b'\xa2\xcc\x87\x00\x01\x94\x98\x93Y\x90X\xa9\x97\x9c\x9fkebb\x0c\x10@\x83\xd3\x87\x00\x00\x00\x00\xff\xff'

(6/7) I havent answered chrome for a while, so it sended a RST_STREAM frame to cancel the previous SYN_STREAM request of (3) (this is OK)

What is the problem with the decompress function?

or does anyone know how decompress the second SYN_STREAM's NV block?

Thanks!

edit:

I add a full example of logs.

Thanks to @Mark, I see .flush() was not called, but the main error was 'incorrect header')

Answer 1

You need to flush. E.g.:

temp = decompressor.decompress(buf)
return temp + decompressor.flush()

Even so, the data you provided is not a valid zlib stream. Is it perhaps a continuation of the the previous stream?