Unable to connect Tight VNC Viewer using SSL

Question

Ive downloaded the Enhanced Tight VNC Viewer (http://www.karlrunge.com/x11vnc/ssvnc.html) and followed instructions to connect the viewer on my local machine(running Win 7) to another machine running the x11vnc server.

The remote machine is running ubuntu 16.04 and has x11vnc server installed (v0.9.13). I am able to connect to it without the ssl option but it fails when I use SSL.

Steps followed

This dialog helps you to create a simple Self-Signed SSL certificate.  

    On Unix the openssl(1) program must be installed and in $PATH.
    On Windows, a copy of the openssl program is provided for convenience.

    The resulting certificate files can be used for either:

       1) authenticating yourself (VNC Viewer) to a VNC Server
    or 2) your verifying the identity of a remote VNC Server.

    In either case you will need to safely copy one of the generated key or
    certificate files to the remote VNC Server and have the VNC Server use
    it.  Or you could send it to the system administrator of the VNC Server.

    For the purpose of description, assume that the filename selected in the
    "Save to file" entry is "vnccert.pem".  That file will be generated
    by this process and so will the "vnccert.crt" file.  "vnccert.pem"
    contains both the Private Key and the Public Certificate.  "vnccert.crt"
    only contains the Public Certificate.

    For case 1) you would copy "vnccert.crt" to the VNC Server side and 
    instruct the server to use it.  For x11vnc it would be for example:

        x11vnc -sslverify /path/to/vnccert.crt -ssl SAVE ...

    (it is also possible to handle many client certs at once in a directory,
    see the -sslverify documentation).  Then you would use "vnccert.pem"
    as the MyCert entry in the SSL Certificates dialog.

    For case 2) you would copy "vnccert.pem" to the VNC Server side and 
    instruct the server to use it.  For x11vnc it would be for example:

        x11vnc -ssl /path/to/vnccert.pem

    Then you would use "vnccert.crt" as the as the ServerCert entry in the
    "SSL Certificates" dialog.


    Creating the Certificate:

    Choose a output filename (ending in .pem) in the "Save to file" entry.

    Then fill in the identification information (Country, State or Province,
    etc).

    The click on "Create" to generate the certificate files.

    Encrypting the Private Key:  It is a very good idea to encrypt the
    Private Key that goes in the "vnccert.pem".  The downside is that
    whenever that key is used (e.g. starting up x11vnc using it) then
    the passphrase will need to be created.  If you do not encrypt it and
    somebody steals a copy of the "vnccert.pem" file then they can pretend
    to be you.

    After you have created the certificate files, you must copy and import
    either "vnccert.pem" or "vnccert.pem" to the remote VNC Server and
    also select the other file in the "SSL Certificates" dialog.
    See the description above.

    For more information see:

           http://www.karlrunge.com/x11vnc/ssl.html
           http://www.karlrunge.com/x11vnc/faq.html#faq-ssl-tunnel-int

    The first one describes how to use x11vnc to create Certificate
    Authority (CA) certificates in addition to Self-Signed ones.


    Tip: if you choose the "Common Name" to be the internet hostname
    (e.g. gateway.mydomain.com) that connections will be made to or
    from that will avoid many dialogs when connecting mentioning that
    the hostname does not match the Common Name.

So, I've followed steps for Option 1 i.e., to authenticate VNC Viewer to a VNC Server.

Started the server using the command

• x11vnc -display :0 -sslverify ~/vnccert2.crt -ssl SAVE

Client Side:

• Provided the path of the pem file generated in the client system to
  the viewer (vnccert2.pem file) in MyCert field

• Clicked Connect with option Use SSL selected

Saved the certificate received from the remote server into Accepted Certs dir

The logs at the server are as below

16/07/2018 16:28:34 SSL: accept_openssl(OPENSSL_VNC)
16/07/2018 16:28:34 SSL: spawning helper process to handle: 10.221.49.127:56668
16/07/2018 16:28:34 SSL: helper for peerport 56668 is pid 17094:
16/07/2018 16:28:34 connect_tcp: trying:   127.0.0.1 20000
16/07/2018 16:28:34 check_vnc_tls_mode: waited: 0.000008 / 1.40 input: SSL Handshake
16/07/2018 16:28:34 SSL: ssl_init[17094]: 12/12 initialization timeout: 20 secs.
16/07/2018 16:28:34 SSL: ssl_helper[17094]: SSL_accept() *FATAL: -1 SSL FAILED
16/07/2018 16:28:34 SSL: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
16/07/2018 16:28:34 SSL: ssl_helper[17094]: Proto: TLSv1
16/07/2018 16:28:34 SSL: ssl_helper[17094]: exit case 2 (ssl_init failed)
16/07/2018 16:28:34 SSL: accept_openssl: cookie from ssl_helper[17094] FAILED. 0
16/07/2018 16:28:39 SSL: accept_openssl(OPENSSL_VNC)
16/07/2018 16:28:39 SSL: spawning helper process to handle: 10.221.49.127:56670
16/07/2018 16:28:39 SSL: helper for peerport 56670 is pid 17095:
16/07/2018 16:28:39 connect_tcp: trying:   127.0.0.1 20000
16/07/2018 16:28:39 check_vnc_tls_mode: waited: 0.000013 / 1.40 input: SSL Handshake
16/07/2018 16:28:39 SSL: ssl_init[17095]: 12/12 initialization timeout: 20 secs.
16/07/2018 16:28:39 SSL: ssl_helper[17095]: SSL_accept() *FATAL: -1 SSL FAILED
16/07/2018 16:28:39 SSL: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
16/07/2018 16:28:39 SSL: ssl_helper[17095]: Proto: nosession
16/07/2018 16:28:39 SSL: ssl_helper[17095]: exit case 2 (ssl_init failed)
16/07/2018 16:28:39 SSL: accept_openssl: cookie from ssl_helper[17095] FAILED. 0

I'm not sure where I am going wrong as the certificate at the client side exists but still the server throws the message "ssl3_get_client_certificate:peer did not return a certificate" also after this it throws another error "ssl3_get_client_hello:wrong version number"

Answer 1

Have you checked the ports used on both attempts? SSL connections may require a different port.

Also, I've found this article on the ServerFault site. The accepted response stands:

The stunnel server has options = NO_SSLv3, but the client is trying to connect using SSLv3. You need to upgrade the client to support a newer version of SSL or you need to change the stunnel configuration to accept SSLv3.