I am running kafka-topics.sh :9098 --describe --topic __consumer_offsets --command-config /etc/client.properties
its throwing below error
Failed to create new KafkaAdminClient
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:541)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:137)
at org.apache.kafka.tools.TopicCommand$TopicService.createAdminClient(TopicCommand.java:437)
at org.apache.kafka.tools.TopicCommand$TopicService.<init>(TopicCommand.java:426)
at org.apache.kafka.tools.TopicCommand.execute(TopicCommand.java:98)
at org.apache.kafka.tools.TopicCommand.mainNoExit(TopicCommand.java:87)
at org.apache.kafka.tools.TopicCommand.main(TopicCommand.java:82)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:252)
at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:189)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:525)
... 7 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/client/certs/keystore.bcfks of type BCFKS
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
... 9 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/client/certs/keystore.bcfks of type BCFKS
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:382)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:354)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:304)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:164)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180)
... 13 more
Caused by: java.security.KeyStoreException: BCFKS not found
at java.base/java.security.KeyStore.getInstance(KeyStore.java:878)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:376)
... 19 more
Caused by: java.security.NoSuchAlgorithmException: BCFKS KeyStore not available
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
at java.base/java.security.Security.getImpl(Security.java:656)
at java.base/java.security.KeyStore.getInstance(KeyStore.java:875)
... 20 more
my client.properties file contains
cat client.properties
# Kafka client configuration
bootstrap.servers=xxxx.amazonaws.com
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
# SSL configurations for BouncyCastle
ssl.truststore.type=BCFKS
ssl.truststore.location=/etc/client/certs/truststore.bcfks
ssl.truststore.password=<redacted>
ssl.keystore.type=BCFKS
ssl.keystore.location=/etc/client/certs/keystore.bcfks
ssl.keystore.password=<redacted>
# Configure the BouncyCastle provider
ssl.security.provider=BouncyCastleProvider
Also I have set java.security file as
cat java.security | grep security.provider
# security.provider.<n>=<provName | className>
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJSSE
security.provider.7=SunJCE
security.provider.8=SunJGSS
security.provider.9=SunSASL
security.provider.10=XMLDSig
security.provider.11=SunPCSC
security.provider.12=JdkLDAP
security.provider.13=JdkSASL
security.provider.14=SunPKCS11
# jdk.security.provider.preferred=AES/GCM/NoPadding:SunJCE, \
#jdk.security.provider.preferred=
login.configuration.provider=sun.security.provider.ConfigFile
policy.provider=sun.security.provider.PolicyFile
# provider (sun.security.provider.PolicyFile) does not support this property.
root@kafka-lag-dp-report-5254-7459f94c7d-xpjxl:/opt/java/openjdk/lib/security# cat java.security | grep fips
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
fips.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
fips.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
scratching my head what else I am missing. Please help me identify the issue causing this error.