Python urllib2 slow upload

525 Views Asked by At

Urllib for python seems to be incedibly slow at uploading a file (using multipart/form-data)

The browser (Chrome) does it in under 20 seconds, while the script uses almost i minute for the same file.

I'm using urllib2 for the connection, and poster to create the http headers and data, the version of python in question is 2.7.

def upSong(fileName):
    datagen, headers = multipart_encode({"mumuregularfile_0": open(fileName, "rb")})

    uploadID = math.floor(random.random()*1000000)
    request = urllib2.Request("http://upload0.mumuplayer.com:443/?browserID=" + browserID + "&browserUploadID=" + str(uploadID), datagen, headers)

    urllib2.urlopen(request).read()

Is there a way to speed up pythons/urllibs connection, or is this just a limitation of the python language?

EDIT: it should be noted that i already tested all parts, and it is without a doubt urllib.read()

2

There are 2 best solutions below

1
On BEST ANSWER

Chromium probably used compression (if supported by the website), while urllib does not look like using it (grepping "gz" or "bz" gives no result).

I am not sure about it, but Chromium may also be more optimized than traditional connections, using socket hacks or something...

2
On

you may add some time-reporter functions after lines. by this you could be able to pin point which step is lowering the speed or taking longer time. it may also vary on network speed ...