JavaMail (SMTPS) + Grizzly, handshake failure

Question

I'm currently trying to test my own implementation of SMTP server which i made on top of Grizzly framework and now struggling with handshake problem during checking SMTPS support with help of JavaMail.
I found similar problem with handshake but hopefully (or not for me) i don't use client mode for Grizlly SSLEngineConfigurator, so i guess that root of the problem should be different.
My code for Grizzly SSL configuration initialization (in integration test) looks following:

try (InputStream keystoreStream = TransportEncryptionIT.class.getResourceAsStream("server_keystore.jks")) {
     SSLContextConfigurator sslCon = new SSLContextConfigurator();
     sslCon.setSecurityProtocol("TLSv1.2");
     sslCon.setKeyStoreType("JKS");
     sslCon.setKeyStoreBytes(IOUtils.toByteArray(keystoreStream));
     sslCon.setKeyStorePass("");
     sslConf = new SSLEngineConfigurator(sslCon, false, false, false);
}

I also install SSLFilter to FilterChain (with server SSL config) immidiatly after TransportFilter (which is the first filter in chain).

filterChainBuilder.add(new SSLFilter(sslConf , null));

And configure javamail for SMTPS in following way:

Properties properties = new Properties();
properties.setProperty("mail.transport.protocol", "smtps");
properties.setProperty("mail.smtps.host", ${host});
properties.setProperty("mail.smtps.port", ${port});
//need this since i'm using self signed certificate which is not added to truststore
properties.setProperty("mail.smtps.ssl.trust", "*"); 

Results of handshake (SSL debug server):

[Raw read]: length = 163
0000: 01 00 00 9F 03 01 54 7C   D5 30 98 30 CA 59 39 D7  ......T..0.0.Y9.
0010: B0 4A B6 FC 8F 8D E9 BD   4B 88 D6 BF EE E8 F9 FF  .J......K.......
0020: 8A EB 28 CF 98 AB 00 00   38 C0 0A C0 14 00 35 C0  ..(.....8.....5.
0030: 05 C0 0F 00 39 00 38 C0   09 C0 13 00 2F C0 04 C0  ....9.8...../...
0040: 0E 00 33 00 32 C0 07 C0   11 00 05 C0 02 C0 0C C0  ..3.2...........
0050: 08 C0 12 00 0A C0 03 C0   0D 00 16 00 13 00 04 00  ................
0060: FF 01 00 00 3E 00 0A 00   34 00 32 00 17 00 01 00  ....>...4.2.....
0070: 03 00 13 00 15 00 06 00   07 00 09 00 0A 00 18 00  ................
0080: 0B 00 0C 00 19 00 0D 00   0E 00 0F 00 10 00 11 00  ................
0090: 02 00 12 00 04 00 05 00   14 00 08 00 16 00 0B 00  ................
00A0: 02 01 00                                           ...
Grizzly-worker(1), READ: TLSv1 Handshake, length = 163
Grizzly-worker(2), fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 1
Grizzly-worker(2), SEND TLSv1.2 ALERT:  fatal, description = internal_error
Grizzly-worker(2), WRITE: TLSv1.2 Alert, length = 2

Results of handshake (SSL debug client):

[Raw read]: length = 249
0000: 01 00 00 F5 03 03 54 7C   D5 30 6E 1C E7 B3 36 DE  ......T..0n...6.
0010: A6 26 73 78 2B 66 D1 D1   E8 C4 94 CA 63 34 22 BF  .&sx+f......c4".
0020: 60 9D 13 03 59 1F 00 00   70 C0 24 C0 28 00 3D C0  `...Y...p.$.(.=.
0030: 26 C0 2A 00 6B 00 6A C0   0A C0 14 00 35 C0 05 C0  &.*.k.j.....5...
0040: 0F 00 39 00 38 C0 23 C0   27 00 3C C0 25 C0 29 00  ..9.8.#.'.<.%.).
0050: 67 00 40 C0 09 C0 13 00   2F C0 04 C0 0E 00 33 00  g.@...../.....3.
0060: 32 C0 07 C0 11 00 05 C0   02 C0 0C C0 2C C0 2B C0  2...........,.+.
0070: 30 00 9D C0 2E C0 32 00   9F 00 A3 C0 2F 00 9C C0  0.....2...../...
0080: 2D C0 31 00 9E 00 A2 C0   08 C0 12 00 0A C0 03 C0  -.1.............
0090: 0D 00 16 00 13 00 04 00   FF 01 00 00 5C 00 0A 00  ............\...
00A0: 34 00 32 00 17 00 01 00   03 00 13 00 15 00 06 00  4.2.............
00B0: 07 00 09 00 0A 00 18 00   0B 00 0C 00 19 00 0D 00  ................
00C0: 0E 00 0F 00 10 00 11 00   02 00 12 00 04 00 05 00  ................
00D0: 14 00 08 00 16 00 0B 00   02 01 00 00 0D 00 1A 00  ................
00E0: 18 06 03 06 01 05 03 05   01 04 03 04 01 03 03 03  ................
00F0: 01 02 03 02 01 02 02 01   01                       .........
main, READ: TLSv1.2 Handshake, length = 249
main, handling exception: javax.net.ssl.SSLProtocolException: Handshake message sequence     violation, 1
main, SEND TLSv1.2 ALERT:  fatal, description = unexpected_message
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 0A                               .......
main, called closeSocket()

---

Strange thing for me is that SMTP + StartTLS works perfectly with my server, handshake was done successfully, i checked this via ssl debug.

Sole thing which I'm not sure is that I'm using self created certificate which is not added to truststore. Could it be that java mail option "mail.${protocol}.ssl.trust" works for SMTP + StartTLS but somehow does not work for SMTPS ?
I also tried to use MailSSLSocketFactory to force javamail to trust server certificate but it seems to be even not invoked (checked with debuger).

JavaMail version 1.4.7
Grizzly 2.3.17

---

Update

Finally i found difference in source code, it seems to be nothing about javamail but my usage of Grizzly framework and installation of SSLFilter to FilterChain. When I start server as SMTPS server, filter is installed in following way:

    filterChainBuilder.add(new TransportFilter());

    if (configuration.getSslConfig() != null) {
        sslFilter = new SSLFilter(configuration.getSslConfig(), null);
        if (configuration.isSmtpsEnabled()) {
            filterChainBuilder.add(sslFilter);
        }
    }

    smtpLineCodecFilter = new SmtpLineCodecFilter();
    filterChainBuilder.add(smtpLineCodecFilter);

But for StartTLS (since client might ask to raise transport encryption) it's done in another way:

    FilterChain securedFilterChain = new DefaultFilterChain(defaultChain);
    int transportFilterindex = defaultChain.indexOfType(TransportFilter.class);
    // Add connection security layer to the chain
    securedFilterChain.add(transportFilterindex + 1, sslFilter);

This difference leads to different filters to be installed, so filter chain looks like this:
In first case - SSLBaseFilter.SSLTransportFilterWrapper -> SSLFilter
In second case - TransportFilter -> SSLFilter

Now I don't know for sure which is the proper way, but at least second case works for me.
I would highly appreciate Grizzly Guru answers :)

Answer 1

Finally i found reason for handshake failure.

It was all about printing SMTP server banner at handleAccept method.
It's ok for plain SMTP connection and StartTLS case since there is no handshake imidiately follows after client esteblished connection. But for SMTPS it won't work in this way, since client expects server helo and not the welcome banner.
Really stupid mistake :/

Link to repository with source code

Answer 2

You're configuring everything for the "smtps" protocol, but perhaps you're not actually using the smtps protocol? How are you connecting to the server? Try using getTransport("smtps"), as in the code example in this JavaMail FAQ entry. Also, make sure you're not using Session.getDefaultInstance. If that doesn't work, post the JavaMail debug output.