I'm using Red5 1.0.3, and trying to accomplish RTMPS over port 8443, with a self-signed certificate.
What I've done already:
- Enabled RTMPS in conf/red5-core.xml
- Added my self-signed certificate to the keystore and truststore
- Confirmed that the passphrase for the keystore and truststore is correct (this is a test, so the passphrases are the same for the sake of simplicity)
- Resolved the "untrusted certificate" issue by accepting the exception in my browser (using Chrome on Ubuntu)
- Ensured that Red5 was listening on 8443 and the port was connectable
The next roadblock I have is an "empty response" in the browser when trying to connect to Red5.
From the Red5 log:
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmps.RTMPSMinaIoHandler - SSL provider is: SunJSSE version 1.7
[DEBUG] [NioProcessor-30] org.red5.server.BaseConnection - New BaseConnection - type: persistent
[DEBUG] [NioProcessor-30] org.red5.server.BaseConnection - Generated session id: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - startWaitForHandshake - 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPMinaIoHandler - Session opened: 77 id: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPMinaIoHandler - Session closed: 77 id: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.BaseRTMPHandler - connectionClosed: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - close: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - State: connect
[DEBUG] [NioProcessor-30] org.red5.server.api.Red5 - Set connection: 7DTVIWZ5UXILR with thread: NioProcessor-30
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - Stream service was not found for scope: null or non-existant
[DEBUG] [NioProcessor-30] org.red5.server.BaseConnection - Close, not connected nothing to do
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - Shutting down scheduler
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - Scheduler - shutdown: true queued: 0
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - Shutting down executor
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPConnection - Executor - shutdown: true queued: 0
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPMinaConnection - IO Session closing: true
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.RTMPMinaConnection - Connection state: RTMP [state=disconnecting, encrypted=false, readChunkSize=128, writeChunkSize=128, encoding=AMF0]
[DEBUG] [NioProcessor-30] org.red5.server.net.rtmp.BaseRTMPHandler - connectionClosed: 7DTVIWZ5UXILR
[DEBUG] [NioProcessor-30] org.red5.server.api.Red5 - Set connection: null with thread: NioProcessor-30
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmps.RTMPSMinaIoHandler - SSL provider is: SunJSSE version 1.7
[DEBUG] [NioProcessor-31] org.red5.server.BaseConnection - New BaseConnection - type: persistent
[DEBUG] [NioProcessor-31] org.red5.server.BaseConnection - Generated session id: UB7GI0V7POCE2
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.RTMPConnection - startWaitForHandshake - UB7GI0V7POCE2
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.RTMPMinaIoHandler - Session opened: 78 id: UB7GI0V7POCE2
[DEBUG] [NioProcessor-31] org.red5.server.api.Red5 - Get connection on thread: NioProcessor-31
[DEBUG] [NioProcessor-31] org.red5.server.api.Red5 - Set connection: UB7GI0V7POCE2 with thread: NioProcessor-31
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.codec.RTMPProtocolDecoder - decodeHandshake - state: RTMPDecodeState [sessionId=UB7GI0V7POCE2, decoderState=0, decoderBufferAmount=0] buffer: HeapBuffer[pos=0 lim=377 cap=1536: 50 4F 53 54 20 2F 6F 70 65 6E 2F 31 20 48 54 54...]
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.codec.RTMPProtocolDecoder - Handshake init too small, buffering. remaining: 377
[DEBUG] [NioProcessor-31] org.red5.server.api.Red5 - Set connection: null with thread: NioProcessor-31
[WARN] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Closing UB7GI0V7POCE2, due to long handshake. State: connect
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - close: UB7GI0V7POCE2
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - State: connect
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.api.Red5 - Set connection: UB7GI0V7POCE2 with thread: RTMPConnectionExecutor#14164738195671
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Stream service was not found for scope: null or non-existant
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.BaseConnection - Close, not connected nothing to do
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Shutting down scheduler
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Scheduler - shutdown: true queued: 0
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Shutting down executor
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnection - Executor - shutdown: true queued: 0
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPMinaConnection - IO Session closing: false
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.RTMPMinaIoHandler - Session closed: 78 id: UB7GI0V7POCE2
[DEBUG] [NioProcessor-31] org.red5.server.net.rtmp.BaseRTMPHandler - connectionClosed: UB7GI0V7POCE2
[DEBUG] [NioProcessor-31] org.red5.server.api.Red5 - Set connection: null with thread: NioProcessor-31
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPMinaConnection - Connection close future: org.apache.mina.core.future.DefaultCloseFuture@41fb8175
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPMinaConnection - Connection is closed
[WARN] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPConnManager - Connection not found for UB7GI0V7POCE2
[DEBUG] [RTMPConnectionExecutor#14164738195671] org.red5.server.net.rtmp.RTMPMinaConnection - Connection state: RTMP [state=disconnected, encrypted=false, readChunkSize=128, writeChunkSize=128, encoding=AMF0]
The connection simply closes. On closer inspection with Wireshark, I discovered an Encrypted Alert (21), which from my understanding means a failed decryption. No hints yet as to why it failed, but it did.
In the browser (flash client), the console outputs
POST https://10.32.1.218:8443/open/1 net::ERR_EMPTY_RESPONSE 10.32.1.218:8443/open/1:1
DEBUG: Flash says: NetworkManager: NetConnection.Connect.Failed
After reading Red5 and RTMPS self-signed certificate, I proceeded to use a real certificate (from StartCom), but when connecting to Red5 with that certificate, I receive
javax.net.ssl.SSLHandshakeException: SSL handshake failed.
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
To which I attempted to re-add the cert to the keystore and truststore with -keyalg RSA (to be explicit), but this did not help.
So back to self-signed certificates....for now anyway.
Advice? Perhaps the client does not accept untrusted certificates and times out on the handshake? Any new paths to look down would be appreciated.
Update
After updating the Flash client, and setting the NetConnection property proxyType = "best", Flash now outputs NetworkManager: NetConnection.Connect.SSLHandshakeFailed
Update II
I've realized that I added the certificates to the keystore and truststore incorrectly. I had added the keys to the keystores without also adding their private keys.
To do this, I used the following answer
I also moved away from a self signed certificate to a CA signed certificate. Issues with self signed weren't worth solving for my situation.