Debug Twill Exception Error

Question

I have been failing to go to a URL usign twill and I couldn't find a way to debug the issue further. I have enabled debug levels in twill for "http","equiv-refresh" and "commands", still twill is not giving any details about the error. Here is the output from twill-sh:

    $ twill-sh

     -= Welcome to twill! =-

    current page:  *empty page*
    >> debug equiv-refresh 1
    DEBUG: setting equiv-refresh debugging to level 1
    current page:  *empty page*
    >> debug http 1
    DEBUG: setting http debugging to level 1
    current page:  *empty page*
    >> debug commands 1
    DEBUG: setting commands debugging to level 1
    current page:  *empty page*
    >> go https://auth.nbnco.net.au/okta/login

    ERROR: cannot go to 'https://auth.nbnco.net.au/okta/login'

    current page:  *empty page*

And here is the output from the python script:

    $ ./test.py
    Traceback (most recent call last):
      File "./test.py", line 13, in <module>
        go("https://auth.nbnco.net.au/okta/login")
      File "/usr/lib/python2.7/site-packages/twill/commands.py", line 109, in go
        browser.go(url)
      File "/usr/lib/python2.7/site-packages/twill/browser.py", line 91, in go
        raise TwillException("cannot go to '%s'" % (url,))
    twill.errors.TwillException: cannot go to 'https://auth.nbnco.net.au/okta/login'

Answer 1

Guess: Twill won't load the page because there is a problem verifying the page's SSL certificate.

Trying to fetch the page using urllib2.urlopen fails with this error:

>>> urllib2.urlopen('https://auth.nbnco.net.au/okta/login')
Traceback (most recent call last):
...                                                                                                       
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>   

Fetching the page using curl produces this output:

$  curl https://auth.nbnco.net.au/okta/login > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

The page loads without error for me in Firefox and Chromium, so the certificate is treated differently by the browsers' certificate handling.

The sites SSL certificate is issued by Symantec. It's possible that this behaviour relates to problems with Symantec certificate issuance in the past which has caused Chrome and Mozilla to announce that they will distrust SSL certificates from Symantec in 2018.

I don't think there is anything you can do in Twill to work around this problem.