NSQ cluster in Kubernetes

Question

I'm trying to set up an NSQ cluster in Kubernetes and having issues. Basically, I want to scale out NSQ and NSQ Lookup. I have a stateful set(2 nodes) definition for both of them. To not post the whole YAML file, I'll post only part of it for NSQ

NSQ container template

command:
        - /nsqd
        - -data-path
        - /data
        - -lookupd-tcp-address
        - nsqlookupd.default.svc.cluster.local:4160

here nsqlookupd.default.svc.cluster.local is a K8s headless service, by doing this I'm expecting the NSQ instance to open a connection with all of the NSQ Lookup instances, which in fact is not happening. It just opens a connection with a random one. However, if I explicitly list all of the NSQ Lookup hosts like this, it works.

command:
            - /nsqd
            - -data-path
            - /data
            - -lookupd-tcp-address
            - nsqlookupd-0.nsqlookupd:4160
            - -lookupd-tcp-address
            - nsqlookupd-1.nsqlookupd:4160

I also wanted to use the headless service DNS name in --broadcast-address for both NSQ and NSQ Lookup but that doesn't work as well.

I'm using nsqio go library for Publishing and Consuming messages, and it looks like I can't use headless service there as well and should explicitly list NSQ/NSQ Lookup pod names when initializing a consumer or publisher.

Am I using this in the wrong way? I mean I want to have horizontally scaled NSQ and NSQLookup instances and not hardcode the addresses.

Answer 1

Try using the below config yaml to deploy it into the K8s cluster. Or else you should checkout the Official Helm : https://github.com/nsqio/helm-chart

Using headless service you can discover all pod IPs.

---

apiVersion: v1
kind: Service
metadata:
  name: nsqlookupd
  labels:
    app: nsq
spec:
  ports:
  - port: 4160
    targetPort: 4160
    name: tcp
  - port: 4161
    targetPort: 4161
    name: http
  publishNotReadyAddresses: true
  clusterIP: None
  selector:
    app: nsq
    component: nsqlookupd
---
apiVersion: v1
kind: Service
metadata:
  name: nsqd
  labels:
    app: nsq
spec:
  ports:
  - port: 4150
    targetPort: 4150
    name: tcp
  - port: 4151
    targetPort: 4151
    name: http
  clusterIP: None
  selector:
    app: nsq
    component: nsqd
---
apiVersion: v1
kind: Service
metadata:
  name: nsqadmin
  labels:
    app: nsq
spec:
  ports:
  - port: 4170
    targetPort: 4170
    name: tcp
  - port: 4171
    targetPort: 4171
    name: http
  selector:
    app: nsq
    component: nsqadmin
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nsqlookupd
spec:
  serviceName: "nsqlookupd"
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nsq
        component: nsqlookupd
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nsq
              - key: component
                operator: In
                values:
                - nsqlookupd
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nsqlookupd
        image: nsqio/nsq:v1.1.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 30m
            memory: 64Mi
        ports:
        - containerPort: 4160
          name: tcp
        - containerPort: 4161
          name: http
        livenessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 5
        readinessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 2
        command:
          - /nsqlookupd
      terminationGracePeriodSeconds: 5
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nsqd
spec:
  serviceName: "nsqd"
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nsq
        component: nsqd
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nsq
              - key: component
                operator: In
                values:
                - nsqd
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nsqd
        image: nsqio/nsq:v1.1.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 30m
            memory: 64Mi
        ports:
        - containerPort: 4150
          name: tcp
        - containerPort: 4151
          name: http
        livenessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 5
        readinessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 2
        volumeMounts:
        - name: datadir
          mountPath: /data
        command:
          - /nsqd
          - -data-path
          - /data
          - -lookupd-tcp-address
          - nsqlookupd-0.nsqlookupd:4160
          - -lookupd-tcp-address
          - nsqlookupd-1.nsqlookupd:4160
          - -lookupd-tcp-address
          - nsqlookupd-2.nsqlookupd:4160
          - -broadcast-address
          - $(HOSTNAME).nsqd
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      terminationGracePeriodSeconds: 5
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes:
        - "ReadWriteOnce"
      storageClassName: ssd
      resources:
        requests:
          storage: 1Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nsqadmin
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nsq
        component: nsqadmin
    spec:
      containers:
      - name: nsqadmin
        image: nsqio/nsq:v1.1.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 30m
            memory: 64Mi
        ports:
        - containerPort: 4170
          name: tcp
        - containerPort: 4171
          name: http
        livenessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 10
        readinessProbe:
          httpGet:
            path: /ping
            port: http
          initialDelaySeconds: 5
        command:
          - /nsqadmin
          - -lookupd-http-address
          - nsqlookupd-0.nsqlookupd:4161
          - -lookupd-http-address
          - nsqlookupd-1.nsqlookupd:4161
          - -lookupd-http-address
          - nsqlookupd-2.nsqlookupd:4161
      terminationGracePeriodSeconds: 5
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nsq
spec:
  rules:
    - host: nsq.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: nsqadmin
            servicePort: 4171

Answer 2

you could use statfulset and headless service to achieve this goal