Is there any way to specify the range for the Diffie-Hellman prime using javax.net.ssl.SSLServerSocket?

Question

I'm trying to reproduce the issue that was raised in Java: Why does SSL handshake give 'Could not generate DH keypair' exception?.

I have a log that shows an exception with this cause:

Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1844)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1827)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1753)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:127)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:94)
    ... 27 more
Caused by: java.lang.RuntimeException: Could not generate DH keypair
    at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:136)
    at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:621)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:205)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    ... 30 more
Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive)
    at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DHKeyPairGenerator.java:120)
    at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:658)
    at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:127)
    ... 38 more

I'm using Axis2 to call a web service. I also have the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" in jre/lib/security (but as someone mentioned on the other post this didn't work for them).

For reproducing this issue I'm trying to host a web service in a separate deployment with a similar environment (also using Axis2). This has a custom SSLServerSocketFactory which is creating and configuring the SSLServerSocket.

Using Wireshark (and also javax.net.debug JVM argument) I can see that the cipher suite that is negotiated is TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA but the Prime DH Server Param in the ServerKeyExchange is only 96 bytes (768 bits).

Is there any way I can configure the SSLServerSocket to use a larger prime during the handshake?

I'm guessing the key is being generated by the security provider for RSA from the JRE. Do I need to add my own provider and add it to jre/lib/security/java.security?

Appologies if this is a silly question, I'm new to DHE an Cipher Suites in general. I just need to know enough to get the server test to work (I'm not touching the client implementation).

Answer 1

So with the push in the right direction from EJP I eventually managed to track down exactly where the strength (and subsequently the DH parameters) is determined. It is indeed in the JRE (specifically in the ServerHandshaker).

Using the security providers with an older JRE with the strength limitation (as mentioned in the original post) will not work here. While using other security providers will probably prevent this error client side the strength is already determined before the implementation of the KeyPairGenerator is initialized.

As a side note I didn't bother upgrading the JRE to reproduce this. I just added the BouncyCastle JCE provider with debug info, put a breakpoint in the org.bouncycastle.jcajce.provider.asymmetric.dh.KeyPairGeneratorSpi.initialize(int, SecureRandom) method and manually changed the strength. This was good enough for my tests.