I'm implementing ssl(mtls) security in apache kafka-3.3.1. As I need to secure the zk first I secured it using SSL. It's running successfully on port 2182 but when I try to run zookeeper-shell.sh it throws error.
command used to start zookeeper-shell.sh zookeeper-shell.sh broker:2182 -zk-tls-config-file /usr/local/kafka/config/zookeeper-client.properties
ERROR Unexpected throwable (org.apache.zookeeper.ClientCnxnSocketNetty)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
WARN Session 0x0 for sever broker/100.31.28.100:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: channel for sessionid 0x0 is lost
Error shown in the logs
ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory)
WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
I've created a CA certificate and then created zk.keystore.jks, zk.truststore.jks, zkclient.keystore.jks, zkclient.truststore.jks. Truststores consist of cacert file and keystores have 3 files in it - server's identity, cacert, and signed csr.
Additional config added to zookeeper.properties:
# ssl-security
secureClientPort=2182
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.trustStore.location=/usr/local/kafka/ssl/zk.truststore.jks
ssl.trustStore.password=zookeeperts
ssl.keyStore.location=/usr/local/kafka/ssl/zk.keystore.jks
ssl.keyStore.password=zookeeperks
ssl.clientAuth=need
config in zookeeper-client.properties:
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=/usr/local/kafka/ssl/zkclient.truststore.jks
zookeeper.ssl.truststore.password=zkclientts
zookeeper.ssl.keystore.location=/usr/local/kafka/ssl/zkclient.keystore.jks
zookeeper.ssl.keystore.password=zkclientks
Following steps taken to resolve error:
i) Checked the path of keystore and truststore files ii) Checked the content of keystore and truststore files iii) Checked validity of certificates iv) kept the DNS and CN same v) Tried without adding -ext SAN in csr vi) Tried with java 11 instead of java 17