I have this strange situation,
I have a link that works on all borwsers that I currently have (chrome,IE,firefox),
I tried to crawl the page using scrapy in python. however I get response.status == 400,
I am using tor + polipo to crawl anonymously
response.body is :
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<title>Proxy error: 400 Couldn't parse URL.</title>
</head><body>
<h1>400 Couldn't parse URL</h1>
<p>The following error occurred while trying to access <strong>https://exmpale.com/blah</strong>:<br><br>
<strong>400 Couldn't parse URL</strong></p>
<hr>Generated Thu, 11 Dec 2014 13:55:38 UTC by Polipo on <em>localhost:8123</em>.
</body></html>
I'm just wondering why that should be, could it be that browser can get results but not scrapy?