i'm trying to deploy kafka with strimzi, problem is, its exposing kafka brokers as load balancers and assigning them an external IP. i want kafka brokers to be available internally and exposed through a load balancer only. below is my deployment file.
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.1.0
replicas: 2
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: external
port: 9094
type: loadbalancer
tls: false
config:
offsets.topic.replication.factor: 2
transaction.state.log.replication.factor: 2
transaction.state.log.min.isr: 2
default.replication.factor: 2
min.insync.replicas: 2
inter.broker.protocol.version: "3.1"
storage:
type: ephemeral
zookeeper:
replicas: 2
storage:
type: ephemeral
screenshot of cluster below
as you can see, there are 3 load balancers with external IP's assigned, whereas i wanted it to be one load balancer with an external IP and 2 kafka brokers.
Yes, this behavior is correct based on the Kafka discovery protocol. So first let's understand it-
Now as we know that the Kafka client directly connects to the broker for sending/receiving the records, the load balancer only comes into the picture for the initial connection and it redirects the client to any one of the available Brokers. Now suppose, if we use the load balancer for subsequent connections as well, what would happen- the load balancer would connect the client to any of the available brokers which might or might not have the partition leader with which the client wants to connect. So Kafka handles this thing using the discovery protocol described above.