kafka mTLS - verify error:num=18:self signed certificate

Question

I'm trying to get mTLS working on Kafka and the client connections is not working .. here are the steps:

-- create CA i.e. ca.crt & ca.key i.e. the CA public cert(ca-cert) & private key (ca-key)

openssl \
req -new -x509 \
-keyout ssl/ca-key \
-out ssl/ca-cert \
-days 365 \
-subj "/CN=my-ca-cert" \
-nodes


Step 2: 

Import the ca-cert into server truststore:

export PASSWORD=<pwd>
 
keytool \
-keystore ssl/kafka.server.truststore.jks \
-alias CARoot \
-storepass $PASSWORD \
-importcert \
-file ssl/ca-cert \
-noprompt

Import the ca-cert into client truststore:
     
keytool \
-keystore ssl/kafka.client.truststore.jks \
-alias CARoot \
-storepass $PASSWORD \
-importcert \
-file ssl/ca-cert \
-noprompt

Step 3: Create keystore for both the client & the server

keytool \
-keystore ssl/kafka.client.keystore.jks \
-alias client-cert \
-validity 365 \
-genkey \
-keyalg RSA \
-storepass ${PASSWORD} \
-keypass ${PASSWORD} \
-dname "CN=my-kafka-client.com" \
-ext SAN=DNS:my-kafka-client.com


keytool \
-keystore ssl/kafka.server.keystore.jks \
-alias server-cert \
-validity 365 \
-genkey \
-keyalg RSA \
-storepass ${PASSWORD} \
-keypass ${PASSWORD} \
-dname "CN=my-kafka-server.com" \
-ext SAN=DNS:my-kafka-server.com


Step 4: Create a request to sign client & server certs

keytool \
-keystore ssl/kafka.client.keystore.jks \
-alias client-cert \
-storepass $PASSWORD \
-certreq \
-file ssl/client-cert-request 


keytool \
-keystore ssl/kafka.server.keystore.jks \
-alias server-cert \
-storepass $PASSWORD \
-certreq \
-file ssl/server-cert-request 

Step 5:

use openssl to get the signed client & server certs, the CA is used to sign the certs 

openssl \
x509 -req \
-CA ssl/ca-cert \
-CAkey ssl/ca-key \
-in ssl/client-cert-request \
-out ssl/client-cert-signed \
-days 365 \
-CAcreateserial \
-passin pass:$PASSWORD

openssl \
x509 -req \
-CA ssl/ca-cert \
-CAkey ssl/ca-key \
-in ssl/server-cert-request \
-out ssl/server-cert-signed \
-days 365 \
-CAcreateserial \
-passin pass:$PASSWORD



Step 6:
Import the signed certs & the ca-cert into both the keystores :

Note : 
1. ca-cert needs to be imported into the server & client keystores 
2. server-cert-signed - needs to be imported into kafka.server.keystore.jks
3. client-cert-signed - needs to be imported into kafka.client.keystore.jks

- importing into kafka.client.keystore.jks
keytool \
-keystore ssl/kafka.client.keystore.jks \
-alias CARoot \
-storepass $PASSWORD \
-importcert \
-file ssl/ca-cert \
-noprompt   


- importing into kafka.server.keystore.jks
keytool \
-keystore ssl/kafka.server.keystore.jks \
-alias CARoot \
-storepass $PASSWORD \
-importcert \
-file ssl/ca-cert \
-noprompt   

- importing client-cert-signed into kafka.client.keystore.jks
keytool \
-keystore ssl/kafka.client.keystore.jks \
-alias client \
-storepass $PASSWORD \
-importcert \
-file ssl/client-cert-signed \
-noprompt 


- importing server-cert-signed into kafka.server.keystore.jks
keytool \
-keystore ssl/kafka.server.keystore.jks \
-alias server \
-storepass $PASSWORD \
-importcert \
-file ssl/server-cert-signed \
-noprompt 

Changes in the kafka server.properties :

#MTLS
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093

ssl.keystore.location=ssl/kafka.server.keystore.jks 
ssl.keystore.password=<pwd> 
ssl.key.password=<pwd> 
ssl.truststore.location=ssl/kafka.server.truststore.jks 
ssl.truststore.password=<pwd>

ssl.client.auth=required
#MTLS

When i run the command to test the SSL connection (after starting zookeeper & kafka broker), i get the following error :

Command:  openssl s_client -connect localhost:9093

response :

CONNECTED(00000005)
depth=0 CN = my-my-kafka-server.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = my-my-kafka-server.com
verify return:1
4305255916:error:1401E0F4:SSL routines:CONNECT_CR_FINISHED:unexpected message:/System/Volumes/Data/SWE/macOS/BuildRoots/e90674e518/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.2/libressl-2.8/ssl/ssl_both.c:510:
---
Certificate chain
 0 s:/CN=my-my-kafka-server.com
   i:/CN=my-my-kafka-server.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MI..................................................................
SDP/f4OiQes68QGbJlbJfjX3F+CDlmgJ8XRkwBw0hpPpRb0TMvpXyGaadS3YMB
X7cVU2A=
-----END CERTIFICATE-----
subject=/CN=my-my-kafka-server.com
issuer=/CN=my-my-kafka-server.com
---
Acceptable client certificate CA names
/CN=Karans-MacBook-Pro.local
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 2524 bytes and written 120 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: B09FF86C25C9D1081E04DF27A8B376EE3419167308B5D3C819EF7AA68145D35E
    Session-ID-ctx: 
    Master-Key: 2ECD8BEC9B08A2E0DE14726F3F2B64397338F87A81CED67ECCBCFE7FC185B3E3C6A4951D308A9BD7B09942F705847FAA
    Start Time: 1702274305
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

How do i fix this error ?

CONNECTED(00000005)
depth=0 CN = my-my-kafka-server.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = my-my-kafka-server.com
verify return:1

Note : Kafka broker & client are on the same macbook. Also, what should be the value in CN (should it be the hostname OR can i give it a random string like - my-kafka-server) ? how is the CN used in the connection, wondering of that is causing an issue ?

tia!

Update : error in the broker logs

javax.net.ssl|ERROR|43|data-plane-kafka-network-thread-0-ListenerName(SSL)-SSL-3|2023-12-11 15:38:42.211 PST|TransportContext.java:341|Fatal (BAD_CERTIFICATE): Empty server certificate chain (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Empty server certificate chain
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
    at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
    at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
    at kafka.network.Processor.poll(SocketServer.scala:989)
    at kafka.network.Processor.run(SocketServer.scala:892)
    at java.base/java.lang.Thread.run(Thread.java:829)}