using python-requests with cntlm corporate proxy

Question

I got some python applications I'd like to run behind a corporate proxy. I'm using cntlm and I think it's configured properly so far because wget, curl and pip and so on are working pretty well. i.e.:

sudo wget http://apple.com

Console-Output:

andre@VirtualBox:~$ wget apple.com
--2018-04-04 10:38:55--  http://apple.com/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.apple.com/ [following]
--2018-04-04 10:38:56--  https://www.apple.com/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 45704 (45K) [text/html]
Saving to: ‘index.html.3’

index.html.3        100%[===================>]  44,63K  --.-KB/s    in 0s      

2018-04-04 10:38:56 (312 MB/s) - ‘index.html.3’ saved [45704/45704]

is working - where this:

import requests
url = 'https://apple.com'
session = requests.session()
r = session.get(url)
print(r.text)

Console-Output:

Traceback (most recent call last):
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connection.py", line 294, in connect
    self._tunnel()
  File "/usr/lib/python3.5/http/client.py", line 832, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andre/.local/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/andre/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apple.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andre/PycharmProjects/Notifications/de/seandre/rest/requesttest.py", line 5, in <module>
    r = session.get(url)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/andre/.local/lib/python3.5/site-packages/requests/adapters.py", line 502, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='apple.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))

is not working.

Many thanks in advance!

Answer 1

You should set environment variable HTTP_PROXY

export HTTP_PROXY='http://mydomain\username:[email protected]:3128'

And try this example code.

url = 'https://example.com/example'
req = urllib.request.Request(
url,
data=None,
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'})
proxy = urllib.request.ProxyHandler({'https': os.environ['HTTP_PROXY']})
auth = urllib.request.HTTPBasicAuthHandler()
opener = urllib.request.build_opener(proxy, auth, urllib.request.HTTPHandler)
urllib.request.install_opener(opener)
data = urllib.request.urlopen(req)

But for me very usefull cntlm proxy