Istio Ingress Gateway to serve Strimzi Apache Kafka on Kubernetes

Question

I have a question on how to configure the Strimzi Kafka Operator with an Istio Ingress Gateway to serve both the bootstrap and the broker services.

I have deployed Istio to with following Gateway and Virtual Services to server both the bootstrap and brokers:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
  name: strimzi-kafka-gw-broker
  namespace: strimzi
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - myserver
    port:
      name: https
      number: 9094
      protocol: HTTP2
    tls:
      mode: SIMPLE
      credentialName: myserver-tls
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
  name: strimzi-kafka-gw-bootstrap
  namespace: strimzi
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - myserver
    port:
      name: tls-9093
      number: 9093
      protocol: TLS
    tls:
      mode: SIMPLE
      credentialName: myserver-tls
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
  name: strimzi-kafka-vs-broker
  namespace: strimzi
spec:
  gateways:
  - strimzi-kafka-gw-broker
  hosts:
  - my-server
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: kafka-cluster-01-kafka-brokers
        port:
          number: 9092
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
  name: strimzi-kafka-vs-bootstrap
  namespace: strimzi
spec:
  gateways:
  - strimzi-kafka-gw-bootstrap
  hosts:
  - my-server
  tcp:
  - match:
    - port: 9093
    route:
    - destination:
        host: kafka-cluster-01-kafka-bootstrap
        port:
          number: 9092

The problem is, when I deploy the Kafka resource via Strimzi Kafka operator in Kubernetes, I need to set the advertised address and advertised port for the brokers to the kafka clients to be used via Ingress, it will also create a listener on the same port for Istio not being able to serve the same port.

Stimzi Kafka Resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster-01
spec:
  kafka:
    version: 3.1.0
    replicas: 2
    listeners:
      - name: internal
        port: 9092
        type: internal
        tls: false
        configuration:
          brokers:
          - broker: 0
            advertisedHost: my-server
            advertisedPort: 9094
          - broker: 1
            advertisedHost: my-server
            advertisedPort: 9094
          - broker: 2
            advertisedHost: my-server
            advertisedPort: 9094
    config:
      offsets.topic.replication.factor: 2
      transaction.state.log.replication.factor: 2
      transaction.state.log.min.isr: 2
      default.replication.factor: 2
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.1"
      receive.message.max.bytes: 1513486160
      advertised: my-server
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

How can I tell Kafka (Strimzi) to use use a advertised address and port for the brokers without creating a listener on it to use an Ingress (Istio) in front of it?

I didn't find this on the documentation of Strimzi.

Thank you.

Answer 1

I also circumvent the slowliness now. The problem was, that I used the kubernetes service kafka-cluster-01-kafka-brokers that the Strimzi Operator deployed, but it worked like a loadbalancer between my two kafka brokers, so it sometimes requested the wrong partition in my topic which the requested broker did not hold. After installing a dedicated service per broker pod, I was able to fully get a working kafka queue with Istio in front.

I just need to tell the Stimzi operator somehow, how to deploy a dedicated services per pod and label the pods respectively.