Differences between wget and browser wireshark traffic

Question

I'm trying to get Wireshark output that is as close as possible to using a browser manually, via wget or urllib.

The output is different, and I was wondering why, and how do I overcome this?

Thanks!

Answer 1

wget is used primarily to grab whole or partial web sites for offline viewing, or for fast download of single files from HTTP or FTP servers instead.

A browser request contains HTTP headers like User Agent, Referer, etc.

If you want to mimic wget to a browser like request, you can pass HTTP headers with your wget request.

Something like this-

# wget http://www.remote.co.in/images/myimage.jpg --header="User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" --header="Accept: image/png,image/;q=0.8,/*;q=0.5" --header="Accept-Language: en-US,en;q=0.5" --header="Accept-Encoding: gzip, deflate" --header="Referer: http://www.mywebsite.com"

Answer 2

There are a couple things...

A browser:
May have several specific headers (useragent, cookies, referer, misc. pplugins, no-track) Requests all child elements/scripts/resources, possibly on the same connection (keep-alive) May request gzipped datastream in return

WGet:
Has minimal headers by default (useragent), but can use/alter others with parameters Is generally a 1-off, requesting only the main html only and not its child resources

It may be if you are seeing different main HTML that the site is server-side scripting tailored content based on useragent and/or cookies (e.g. "logged in")