I am building a Hortonworks cluster with both HDP and HDF to be installed. I first installed HDP and then installed/integrated with HDF on top of it.
Environment Details: OS: Red Hat Enterprise Linux Server release 7.6 (Maipo) Version: Ambari -2.7.3, HDP - 3.1, HDF -3.4.0
Basically HDP-3.1 has kafka 1.0.1 in the package and in HDF has kafka 2.1.0 packages are available and I need HDF version of Kafka to be available. Though I had installed Kafka from HDF, Ambari shows the kafka version of 1.0.1. After integration with HDF, it's not showing up the Kafka-2.1.0 in the Add service list.
I need to know, how can I get Kafka 2.1.0 installed in the cluster.
Also, Is it a possibility that version showed is 1.0.1, though Kafka 2.1.0 is installed.
Doubtful. Ambari parses the packages that are installed on the machine to determine the versions
My suggestion would be to manually SSH to each machine and try to install Kafka from yum and see what versions are available.
If you appropriately setup the HDF YUM repos, then that version of Kafka should be available.
Alternatively, you could always install Kafka/Zookeeper externally and manage it outside of Ambari