kubernetess multiple deployments using one code base but different configuration (environement variables)

Question

I have a project where we are consuming data from kafka and publishing to mongo. In fact the code base does only one task, may be mongo to kafka migration, kafka to mongo migration or something else.

we have to consume from different kafka topics and publish to different mongo collections. Now these are parallel streams of work.

Current design is to have one codebase which can consume from Any topic and publish to Any mongo collection which is configurable using Environment variables. So we created One kubernetes Pod and have multiple containers inside it. each container has different environment variables.

My questions:

1. Is it wise to use multiple containers in one pod. Easy to distinguish, but as they are tightly coupled , i am guessing high chance of failure and not actually proper microservice design.
2. Should I create multiple deployments for each of these pipelines ? Would be very difficult to maintain as each will have different deployment configs.
3. Is there any better way to address this ?

Sample of step 1:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations: {}
  name: test-raw-mongodb-sink-apps
  namespace: test-apps
spec:
  selector:
    matchLabels:
      app: test-raw-mongodb-sink-apps
  template:
    metadata:
      labels:
        app: test-raw-mongodb-sink-apps
    spec:
      containers:
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-alchemy
        - name: INPUT_TOPIC
          value: test.raw.ptv.alchemy
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8081"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/dpl/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-alchemy
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-bloomberg
        - name: INPUT_TOPIC
          value: test.raw.pretrade.bloomberg
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8082"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-bloomberg
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-calypso
        - name: INPUT_TOPIC
          value: test.raw.ptv.calypso
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8083"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-calypso
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-dtres
        - name: INPUT_TOPIC
          value: test.raw.ptv.dtres
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8084"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-dtres
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-feds
        - name: INPUT_TOPIC
          value: test.raw.ptv.feds
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8085"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-feds
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-hoops
        - name: INPUT_TOPIC
          value: test.raw.ptv.hoops
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8086"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-hoops
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxcore
        - name: INPUT_TOPIC
          value: test.raw.ptv.murex_core
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8087"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxcore
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxeqd
        - name: INPUT_TOPIC
          value: test.raw.ptv.murex_eqd_sa
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8088"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxeqd
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxgts
        - name: INPUT_TOPIC
          value: test.raw.ptv.murex_gts_sa
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8089"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxgts
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxmr
        - name: INPUT_TOPIC
          value: test.raw.ptv.murex_mr
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8090"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxmr
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxgtscf
        - name: INPUT_TOPIC
          value: test.raw.cashflow.murex_gts_sa
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8091"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxgtscf
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxcoll
        - name: INPUT_TOPIC
          value: test.raw.collateral.mxcoll
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8092"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxcoll
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-mxcoll-link
        - name: INPUT_TOPIC
          value: test.raw.collateral.mxcoll_link
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8093"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-mxcoll-link
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-ost
        - name: INPUT_TOPIC
          value: test.raw.ptv.ost
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8094"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-ost
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      - env:
        - name: EVENTS_TOPIC
          value: test.ops.proc-events
        - name: GROUP_ID
          value: test-mongodb-sink-posmon
        - name: INPUT_TOPIC
          value: test.raw.ptp.posmon
        - name: MONGODB_AUTH_DB
          value: admin
        - name: MONGODB_HOST0
          value: test-mongodb-0.test-mongodb-headless.test-infra
        - name: MONGODB_HOST1
          value: test-mongodb-1.test-mongodb-headless.test-infra
        - name: MONGODB_PASSWORD
          value: test123
        - name: MONGODB_PORT
          value: "27017"
        - name: MONGODB_USERNAME
          value: root
        - name: SERVER_PORT
          value: "8095"
        - name: KAFKA_BROKERS
          value: kafka-cluster-kafka-bootstrap.kafka:9093
        - name: TRUSTSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ca.password
              name: kafka-ca-cert
        - name: KEYSTORE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: user.password
              name: kafka
        image: tools.testCompany.co.za:8093/local/tt--mongodb-map:0.0.7.0-SNAPSHOT
        name: test-mongodb-sink-posmon
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /app/resources
          name: properties
        - mountPath: /stores
          name: stores
          readOnly: true
      

Thanks

Answer 1

A templating tool like Helm will let you fill in the environment-variable values from deploy-time settings. In Helm this would look like:

env:
  - name: EVENTS_TOPIC
    value: {{ .Values.eventsTopic }}
  - name: GROUP_ID
    value: {{ .Values.groupId }}
  - name: INPUT_TOPIC
    value: {{ .Values.inputTopic }}

You could then deploy this multiple times with different sets of topics:

helm install alchemy . \
  --set eventsTopic=test.ops.proc-events \
  --set groupId=test-mongodb-sink-alchemy \
  --set inputTopic=test.raw.ptv.alchemy
helm install bloomberg . \
  --set eventsTopic=test.ops.proc-events \
  --set groupId=test-mongodb-sink-bloomberg \
  --set inputTopic=test.raw.pretrade.bloomberg

You could write the Helm chart to be configured with a list of topic sets, too, and only deploy the set once:

{{- $top := . -}}{{-/* because "range" overwrites "." */-}}
{{- $topic := range $topics -}}
---
apiVersion: v1
kind: Deployment
metadata:
  name: {{ $topic.name }}
spec:
  ...
    env:
      - name: EVENT_TOPIC
        value: {{ $top.Values.eventTopic }}{{/* common to all deployments */}}
      - name: GROUP_ID
        value: test-mongodb-sink-{{ $topic.name }}
      - name: INPUT_TOPIC
        value: {{ $topic.inputTopic }}

Write configuration like:

eventTopic: test.ops.proc-events
topics:
  - name: alchemy
    inputTopic: test.raw.ptv.alchemy
  - name: bloomberg
    inputTopic: test.raw.pretrade.bloomberg

And deploy like:

helm install connector . -f topic-listing.yaml

In any case, you will want only one container per pod. There are a couple of reasons for this. If the list of topics ever changes, this lets you create or delete deployments without interfering with the other topics; if everything was in a single pod, you'd have to stop and restart everything together, and it can take Kafka a minute or two to figure out what happens. In a Kafka context, you can also run as many consumers as there are partitions on a topic, but not really more; if you have a very busy topic you can easily set that deployment's replicas: to have multiple consumers for multiple partitions, but if everything together is in one pod, your only choice is to scale everything together.

Answer 2

Is it wise to use multiple containers in one pod. Easy to distinguish, but as they are tightly coupled , i am guessing high chance of failure and not actually proper microservice design.

You most likely want to deploy them as separate services, so that you can update or re-configure them independently of eachother.

Should I create multiple deployments for each of these pipelines ? Would be very difficult to maintain as each will have different deployment configs.

Kustomize is a built-in tool in kubectl that is a good choice when you want to deploy the same manifest in multiple environments with different configurations. This solution require no additional tool other than kubectl.

Deploying to multiple environments with Kustomize

Directory structure:

base/
  - deployment.yaml      # fully deployable manifest - no templating
  - kustomization.yaml   # default values e.g. for dev environment
app1/
  - kustomization.yaml   # specific values for app1
app2/
  - kustomization.yaml   # specific values for app2

Example Deployment manifest with Kustomization

Here, the environment variables is loaded from a ConfigMap such that we can use configMapGenerator. This file is base/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-sink
  namespace: test-apps
spec:
  template: // some fiels, e.g. labels are omitted in example
    spec:
      containers:
      - name: mongodb-sink
        image: mongodb-map:0.0.7.0-SNAPSHOT
        env:
          - name: MONGODB_HOST0
            value: test-mongodb-0.test-mongodb-headless.test-infra
          - name: MONGODB_HOST1
            value: test-mongodb-1.test-mongodb-headless.test-infra
          - name: GROUP_ID
            valueFrom:
              configMapKeyRef:
                name: my-values
                key: GROUP_ID
          - name: INPUT_TOPIC
            valueFrom:
              configMapKeyRef:
                name: my-values
                key: INPUT_TOPIC
      ...

Also add a base/kustomization.yaml file to describe the configMapGenerator and related files.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml

configMapGenerator:
- name: my-values
  behavior: replace
  literals:
  - GROUP_ID=test-mongodb-sink-calypso
  - INPUT_TOPIC=test.raw.ptv.calypso
  ... # also add your other values

Preview Manifests

kubectl kustomize base/

Apply Manifests

kubectl apply -k base/

Add config for app1 and app2

With app1 we now want to use the manifest we have in base/ and just overlay what is different for app1. This file is app1/kustomization.yaml and similar for app2/kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../base

namePrefix: bloomberg-sink-   # this gives your Deployment a prefixed name

configMapGenerator:
- name: my-values
  behavior: replace
  literals:
  - GROUP_ID=test-mongodb-sink-bloomberg
  - INPUT_TOPIC=test.raw.pretrade.bloomberg
  ... # also add your other values

Preview Manifests

kubectl kustomize app1/

Apply Manifests

kubectl apply -k app1/

Documentation

• Kubernetes: Declarative Management of Kubernetes Objects Using Kustomize
• SIG CLI: Kustomization file