The company I work for requires / impresses that all our selenium test traffic be conducted through https when using the remote selenium grid server that we have hosted in AWS inside a docker instance.
Until now this seems to have been working, but with the most recent version of selenium 3.14. I'm now receiving 100's of "InsecureRequestWarnings" for even the most simple test. I've diagnosed that these are caused by the https connection to selenium grid rather than the test target url it's self, because if I run the same tests from a local selenium server and webdriver I don't get the same issues.
I'm using the following: python 3.6.4, pytest 3.7.2, pytest-selenium 1.13, selenium 3.14 (both locally and on the remote selenium grid), chromedriver 2.41 (both locally and remotely), certifi 2018.8.13, urllib3 1.23
run from various windows boxes (server 2008, windows 10 etc...)
upon running the below code (basically a combination of my conftest.py and a simple login test script) I receive the following warnings (62 of them repeated).
D:\Work\PyTestFramework\VirtEnv3_6\lib\site-packages\urllib3\connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
My sample code:
import pytest
import webdriverwrapper
import webdriverwrapper.wrapper
from webdriverwrapper import DesiredCapabilities
# testtools are just my reusable helper libraries
from testtools import credentials_helper, login_helper, appointment_helper
def pytest_addoption(parser):
parser.addoption('--url', action='store', default='https://bookingportal.mycompany.com.au/OBP',
help='target machine url')
@pytest.fixture(scope='session')
def url(request):
return request.config.getoption('url')
@pytest.fixture(scope='function')
def browser(request):
desired_cap = DesiredCapabilities.CHROME
desired_cap['chromeOptions'] = {}
desired_cap['chromeOptions']['args'] = ['--disable-plugins', '--disable-extensions']
desired_cap['browserName'] = 'chrome'
desired_cap['javascriptEnabled'] = True
hostname = "https://selenium.mygriddocker.com.au/wd/hub"
executor = webdriverwrapper.wrapper.remote.remote_connection.RemoteConnection(hostname, resolve_ip=False)
b = webdriverwrapper.Remote(executor, desired_cap)
request.addfinalizer(lambda *args: b.quit())
return b
@pytest.fixture(scope='function')
def driver(browser, url):
driver = browser
driver.set_window_size(1260, 1080)
driver.get(url)
return driver
# ------------ put test script here
@pytest.mark.usefixtures("driver")
def test_OBP_Login(driver):
testId = 'LogIn01'
credentials_list = credentials_helper.get_csv_data('OBP_LoginDetails.csv', testId)
assert driver.title == 'Booking Portal'
rslt1 = login_helper.login_user(driver, credentials_list)
assert rslt1
rslt2 = appointment_helper.logout_OBP(driver)
assert rslt2
as you can see the address for the selenium grid server I'm calling is https://selenium.mygriddocker.com.au/wd/hub (not the actual real address)
and I have verified that it does have a valid certificate issued by a public authority (Comodo). Note I do not have this issue if I roll back to urllib3 and certifi but I'm not sure at what point it stops working
A lot of people out there have just been suppressing the warnings, but I actually want the certificate to work.
The obvious solution is to follow the links and add certificate verification as they suggest, but the problem is how do I send connection pool manager parameters through the selenium RemoteConnection class?
Suggested fix from https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl :
import certifi
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED', ca_certs=certifi.where())
When I dug into the selenium library I found the use of urllib3.PoolManager was located in the remote_connection module but it did not have any parameters available to send through additional information for the PoolManager when calling it. You can see in my sample code the executor object is where I call the remote_connection (by way of a webdriverwrapper, in regular selenium the path is selenium/webdriver/remote/remote_connection )
The RemoteConnection class has no provision for setting the PoolManager that I could see.
I tried just editing the remote_connection in the hope of hard coding the certificating information into the file in the hope of figuring out how to wrap it later. So I added cert_reqs='CERT_REQUIRED', ca_certs=certifi.where() lines where ever I saw urllib3.PoolManager() (I also imported certifi into that module). By some amazing piece of dumb luck it actually worked. Now I can't figure out how to wrap it.
Has anyone got any other ideas on how to make remote selenium work over https ?
So I've found out that between selenium 3.13 and 3.14 they changed the http request library from httplib to urllib3 and don't seem to have appropriately handled certificates.
So I've discovered they my options are: 1) use selenium 3.13 client side 2) use selenium 3.14 client side and disable InsecureRequestWarnings import urllib3 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
3) use selenium 3.14 client side and hack the remote_connections.py file replacing all instances of urllib3.PoolManager() with urllib3.PoolManager(cert_reqs='CERT_REQUIRED', ca_certs=certifi.where())
the other thing to note is that I think Pycharm caches libraries so if I do a pip update and don't clear cache I git mixed results
4) Somehow wrap remote_connection.py into my own library and import the class adding in the PoolManager parameters - this is beyond my abilities.
This is no longer a question but the info my help some others out there who are receiving the urllib3 insecurewarnings when using https on remote selenium calls.