I am trying to connect SSL site https://link.bollore-logistics.com by using Java 6 and HTMLUnit, I have added more supported Ciphers by using Bouncy Castle. But still, I am receiving handshake failure. I cannot upgrade Java version due to production limitation. Please kindly help
Code of configuration:
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
if (Security.getProvider("BC") == null){
logger.info("Bouncy Castle provider is NOT available");
}
else{
logger.info("Bouncy Castle provider is available");
}
Provider provider = new BouncyCastleProvider();
Security.addProvider(provider);
SSLContext context = SSLContext.getInstance("SSL");
context.init(null, new X509TrustManager[]{new X509TrustManager() {
@Override
public void checkClientTrusted(X509Certificate[] arg0, String arg1) throws CertificateException {
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
@Override
public X509Certificate[] getAcceptedIssuers() {
return null;
}
}}, new SecureRandom());
String[] scs = context.getSocketFactory().getSupportedCipherSuites();
Arrays.sort(scs);
for(String s : scs) {
System.out.println(s);
}
HttpsURLConnection.setDefaultSSLSocketFactory(context.getSocketFactory());
// Create all-trusting host name verifier
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
Code to make connection (HTMLUnit):
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getPage(url);
Here is the error log:
2017-12-06 11:27:40 DEBUG WebClient:325 - Get page for window named '', using WebRequest[<url="http://link.bollore-logistics.com/Contact/SmartSearch/77707575", GET, EncodingType[name=application/x-www-form-urlencoded], [], {Accept-Encoding=gzip, deflate, Accept=*/*}, null>]
2017-12-06 11:27:40 DEBUG WebClient:1259 - Load response for GET http://link.bollore-logistics.com/Contact/SmartSearch/77707575
trigger seeding of SecureRandom
done seeding SecureRandom
2017-12-06 11:27:40 DEBUG RequestAddCookies:122 - CookieSpec selected: mine
2017-12-06 11:27:40 DEBUG RequestAuthCache:76 - Auth cache not set in the context
2017-12-06 11:27:40 DEBUG PoolingHttpClientConnectionManager:249 - Connection request: [route: {}->http://link.bollore-logistics.com:80][total kept alive: 0; route allocated: 0 of 6; total allocated: 0 of 20]
2017-12-06 11:27:40 DEBUG PoolingHttpClientConnectionManager:282 - Connection leased: [id: 0][route: {}->http://link.bollore-logistics.com:80][total kept alive: 0; route allocated: 1 of 6; total allocated: 1 of 20]
2017-12-06 11:27:40 DEBUG MainClientExec:234 - Opening connection {}->http://link.bollore-logistics.com:80
2017-12-06 11:27:40 DEBUG DefaultHttpClientConnectionOperator:131 - Connecting to link.bollore-logistics.com/195.101.158.66:80
2017-12-06 11:27:40 DEBUG DefaultHttpClientConnectionOperator:138 - Connection established 192.168.171.168:52556<->195.101.158.66:80
2017-12-06 11:27:40 DEBUG MainClientExec:255 - Executing request GET /Contact/SmartSearch/77707575 HTTP/1.1
2017-12-06 11:27:40 DEBUG MainClientExec:260 - Target auth state: UNCHALLENGED
2017-12-06 11:27:40 DEBUG MainClientExec:266 - Proxy auth state: UNCHALLENGED
2017-12-06 11:27:40 DEBUG headers:124 - http-outgoing-0 >> GET /Contact/SmartSearch/77707575 HTTP/1.1
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> Host: link.bollore-logistics.com
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> Accept-Language: en-US
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> Accept-Encoding: gzip, deflate
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> Accept: */*
2017-12-06 11:27:40 DEBUG headers:127 - http-outgoing-0 >> Connection: Keep-Alive
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "GET /Contact/SmartSearch/77707575 HTTP/1.1[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "Host: link.bollore-logistics.com[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "Accept-Language: en-US[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "Accept-Encoding: gzip, deflate[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "Accept: */*[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2017-12-06 11:27:40 DEBUG wire:72 - http-outgoing-0 >> "[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "HTTP/1.0 302 Found[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "Location: https://link.bollore-logistics.com/Contact/SmartSearch/77707575[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "Server: BigIP[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "Connection: Keep-Alive[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "Content-Length: 0[\r][\n]"
2017-12-06 11:27:41 DEBUG wire:72 - http-outgoing-0 << "[\r][\n]"
2017-12-06 11:27:41 DEBUG headers:113 - http-outgoing-0 << HTTP/1.0 302 Found
2017-12-06 11:27:41 DEBUG headers:116 - http-outgoing-0 << Location: https://link.bollore-logistics.com/Contact/SmartSearch/77707575
2017-12-06 11:27:41 DEBUG headers:116 - http-outgoing-0 << Server: BigIP
2017-12-06 11:27:41 DEBUG headers:116 - http-outgoing-0 << Connection: Keep-Alive
2017-12-06 11:27:41 DEBUG headers:116 - http-outgoing-0 << Content-Length: 0
2017-12-06 11:27:41 DEBUG MainClientExec:284 - Connection can be kept alive indefinitely
2017-12-06 11:27:41 DEBUG PoolingHttpClientConnectionManager:314 - Connection [id: 0][route: {}->http://link.bollore-logistics.com:80] can be kept alive indefinitely
2017-12-06 11:27:41 DEBUG PoolingHttpClientConnectionManager:320 - Connection released: [id: 0][route: {}->http://link.bollore-logistics.com:80][total kept alive: 1; route allocated: 1 of 6; total allocated: 1 of 20]
2017-12-06 11:27:41 DEBUG WebClient:1349 - Got a redirect status code [302] new location = [https://link.bollore-logistics.com/Contact/SmartSearch/77707575]
2017-12-06 11:27:41 DEBUG WebClient:1259 - Load response for GET https://link.bollore-logistics.com/Contact/SmartSearch/77707575
2017-12-06 11:27:41 DEBUG RequestAddCookies:122 - CookieSpec selected: mine
2017-12-06 11:27:41 DEBUG RequestAuthCache:76 - Auth cache not set in the context
2017-12-06 11:27:41 DEBUG PoolingHttpClientConnectionManager:249 - Connection request: [route: {s}->https://link.bollore-logistics.com:443][total kept alive: 1; route allocated: 0 of 6; total allocated: 1 of 20]
2017-12-06 11:27:41 DEBUG PoolingHttpClientConnectionManager:282 - Connection leased: [id: 1][route: {s}->https://link.bollore-logistics.com:443][total kept alive: 1; route allocated: 1 of 6; total allocated: 2 of 20]
2017-12-06 11:27:41 DEBUG MainClientExec:234 - Opening connection {s}->https://link.bollore-logistics.com:443
2017-12-06 11:27:41 DEBUG DefaultHttpClientConnectionOperator:131 - Connecting to link.bollore-logistics.com/195.101.158.66:443
2017-12-06 11:27:41 DEBUG HtmlUnitSSLConnectionSocketFactory:335 - Connecting socket to link.bollore-logistics.com/195.101.158.66:443 with timeout 90000
2017-12-06 11:27:41 DEBUG HtmlUnitSSLConnectionSocketFactory:388 - Enabled protocols: [TLSv1]
2017-12-06 11:27:41 DEBUG HtmlUnitSSLConnectionSocketFactory:389 - Enabled cipher suites:[SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA]
2017-12-06 11:27:41 DEBUG HtmlUnitSSLConnectionSocketFactory:393 - Starting handshake
%% No cached client session
*** ClientHello, TLSv1
RandomCookie: GMT: 1512530605 bytes = { 147, 117, 177, 152, 172, 87, 109, 103, 219, 130, 24, 140, 216, 108, 154, 3, 162, 211, 205, 89, 29, 221, 91, 103, 85, 39, 206, 187 }
Session ID: {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA]
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
***
[write] MD5 and SHA1 hashes: len = 161
0000: 01 00 00 9D 03 01 5A 27 63 AD 93 75 B1 98 AC 57 ......Z'c..u...W
0010: 6D 67 DB 82 18 8C D8 6C 9A 03 A2 D3 CD 59 1D DD mg.....l.....Y..
0020: 5B 67 55 27 CE BB 00 00 36 00 04 00 05 00 2F C0 [gU'....6...../.
0030: 02 C0 04 C0 0C C0 0E C0 07 C0 09 C0 11 C0 13 00 ................
0040: 33 00 32 00 0A C0 03 C0 0D C0 08 C0 12 00 16 00 3.2.............
0050: 13 00 09 00 15 00 12 00 03 00 08 00 14 00 11 01 ................
0060: 00 00 3E 00 0A 00 34 00 32 00 17 00 01 00 03 00 ..>...4.2.......
0070: 13 00 15 00 06 00 07 00 09 00 0A 00 18 00 0B 00 ................
0080: 0C 00 19 00 0D 00 0E 00 0F 00 10 00 11 00 02 00 ................
0090: 12 00 04 00 05 00 14 00 08 00 16 00 0B 00 02 01 ................
00A0: 00 .
http-bio-8080-exec-3, WRITE: TLSv1 Handshake, length = 161
[Raw write]: length = 166
0000: 16 03 01 00 A1 01 00 00 9D 03 01 5A 27 63 AD 93 ...........Z'c..
0010: 75 B1 98 AC 57 6D 67 DB 82 18 8C D8 6C 9A 03 A2 u...Wmg.....l...
0020: D3 CD 59 1D DD 5B 67 55 27 CE BB 00 00 36 00 04 ..Y..[gU'....6..
0030: 00 05 00 2F C0 02 C0 04 C0 0C C0 0E C0 07 C0 09 .../............
0040: C0 11 C0 13 00 33 00 32 00 0A C0 03 C0 0D C0 08 .....3.2........
0050: C0 12 00 16 00 13 00 09 00 15 00 12 00 03 00 08 ................
0060: 00 14 00 11 01 00 00 3E 00 0A 00 34 00 32 00 17 .......>...4.2..
0070: 00 01 00 03 00 13 00 15 00 06 00 07 00 09 00 0A ................
0080: 00 18 00 0B 00 0C 00 19 00 0D 00 0E 00 0F 00 10 ................
0090: 00 11 00 02 00 12 00 04 00 05 00 14 00 08 00 16 ................
00A0: 00 0B 00 02 01 00 ......
[Raw read]: length = 5
0000: 15 03 01 00 02 .....
[Raw read]: length = 2
0000: 02 28 .(
http-bio-8080-exec-3, READ: TLSv1 Alert, length = 2
http-bio-8080-exec-3, RECV TLSv1 ALERT: fatal, handshake_failure
http-bio-8080-exec-3, called closeSocket()
http-bio-8080-exec-3, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
2017-12-06 11:27:41 DEBUG DefaultManagedHttpClientConnection:87 - http-outgoing-1: Shutdown connection
2017-12-06 11:27:41 DEBUG MainClientExec:128 - Connection discarded
2017-12-06 11:27:41 DEBUG DefaultManagedHttpClientConnection:79 - http-outgoing-1: Close connection
2017-12-06 11:27:41 DEBUG PoolingHttpClientConnectionManager:320 - Connection released: [id: 1][route: {s}->https://link.bollore-logistics.com:443][total kept alive: 1; route allocated: 0 of 6; total allocated: 1 of 20]
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1694)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:939)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1147)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1131)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at com.gargoylesoftware.htmlunit.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:155)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:178)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1313)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1371)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1230)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:338)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:407)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:392)
at com.grabber.Grabber.accessWebpage(Grabber.java:204)
at com.grabber.Grabber.doGet(Grabber.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:619)
I had a similar problem (but not using HtmlUnit in that case. I used a directly a org.apache.http.impl.client.CloseableHttpClient).
It seams there is something wrong with the handshake until java 7 (no problem anymore in java 8). If you try to access a server providing "TLSv1.2" you might get the error reported by you.
If security is not that much of an issue in your case, you can try the to force usage of TLSv1.1 (as long as you have not yet switched to java 8). This shoud be possible this way: