I'm trying to access my local TWiki installation with python http.client. For some reason I always end up with 403 Forbidden. I can access other sub folders in my server, but not twiki. I can access this TWiki page with curl. Is there something special you need to do when accessing /bin/ or /cgi-bin/ folders with python http.client?
Here is example with twiki.org pages, because my localhost is not accessible outside:
>>> import httplib
>>> conn = httplib.HTTPConnection("twiki.org")
>>> conn.request("GET", "/cgi-bin/view/")
>>> r1 = conn.getresponse()
>>> print r1.status, r1.reason
403 Forbidden
>>> data1 = r1.read()
>>> data1
'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don\'t have permission to access /cgi-bin/view/\non this server.</p>\n<hr>\n<address>Apache/2.2.3 (CentOS) Server at twiki.org Port 80</address>\n</body></html>\n'
>>>
I just tried this myself and I found that setting a
User-Agentheader seemed to fix it. It didn't seem to matter what the header was, simply that it was set:Unfortunately I can't shed any light on why Twiki returns a 403 without a
User-Agentheader - I just tried it on the basis that it's one of the likely differences between clients. I assume it's something like the fact that it's trying to decide whether to return the mobile version of the site, but it's really poor not to handle the case of no header gracefully.Hopefully that at least provides a work-around for you, however.
EDIT
Apparently this is part of their default Apache config using the
BrowserMatchNoCasedirective to set an environment variableblockAccesswhich is presumably picked up later to return the observed403 Forbiddenresponse.They seem to think that this prevents DoS attacks somehow, although I'm really unconvinced by anything that can be worked around by simply setting a random
User-Agentstring. As you can tell from that config, they also have a list of "known bad" user agents they attempt to block. You can observe this by attempting to use one of them to fetch from the command-line:I'm sure they have their reasons for doing this, but I must say I'm not impressed.