unable to access mongodb Replicaset pods from other pods | connect EHOSTUNREACH 10.1.231.87:27017

Question

I am running mongodb statefulset in a microk8s kubernetes cluster. I can connect to the PRIMARY replica through my host's terminal using pod's ip address like this:

mongosh "mongodb://10.1.231.87:27017/test" here 10.1.231.87 is the ip address of mongo replica pod. I have another Nodejs pod (in the same default namespace & cluster) which uses a URI string to connect to the mongo replica, but I get the following error:

MongoServerSelectionError: connect EHOSTUNREACH 10.1.231.87:27017 My config file for the statefulset is:

apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongod
spec:
  serviceName: mongodb-service
  replicas: 1
  selector:
    matchLabels:
      role: mongo
  template:
    metadata:
      labels:
        role: mongo
        environment: test
        replicaset: MainRepSet
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongod-container
          image: mongo
          command:
            - "mongod"
            - "--bind_ip"
            - "0.0.0.0"
            - "--replSet"
            - "MainRepSet"
          resources:
            requests:
              cpu: 0.2
              memory: 200Mi
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-volume
              mountPath: /data/db
  volumeClaimTemplates:
   - metadata:
       name: mongo-volume
     spec:
       accessModes: ["ReadWriteOnce"]
       storageClassName: microk8s-hostpath
       resources:
         requests:
           storage: 1Gi

NodeJS pod makes the following call:

mongoose.connect('mongodb://mongod-0.mongodb-service:27017/authDB?replicaSet=MainRepSet&readPreference=primaryPreferred', { useNewUrlParser: true })

Logs of the nodejs pod:

error in connecting to database :  MongooseServerSelectionError: connect EHOSTUNREACH 10.1.231.87:27017
    at NativeConnection.Connection.openUri (/auth_service/node_modules/mongoose/lib/connection.js:824:32)
    at /auth_service/node_modules/mongoose/lib/index.js:412:10
    at /auth_service/node_modules/mongoose/lib/helpers/promiseOrCallback.js:41:5
    at new Promise (<anonymous>)
    at promiseOrCallback (/auth_service/node_modules/mongoose/lib/helpers/promiseOrCallback.js:40:10)
    at Mongoose._promiseOrCallback (/auth_service/node_modules/mongoose/lib/index.js:1265:10)
    at Mongoose.connect (/auth_service/node_modules/mongoose/lib/index.js:411:20)
    at connectAuthDB (/auth_service/dbconnect.js:6:6)
    at Server.<anonymous> (/auth_service/App.js:43:7)
    at Object.onceWrapper (events.js:519:28) {
  reason: TopologyDescription {
    type: 'Unknown',
    servers: Map(1) { 'mongod-0.mongodb-service:27017' => [ServerDescription] },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: null,
    maxElectionId: null,
    maxSetVersion: null,
    commonWireVersion: 0,
    logicalSessionTimeoutMinutes: null
  },
  code: undefined
}

Describing the mongod replica shows:

Name:             mongod-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             mynodename/10.0.208.58
Start Time:       Mon, 28 Nov 2022 13:54:55 +0000
Labels:           controller-revision-hash=mongod-5b69c4bf54
                  environment=test
                  replicaset=MainRepSet
                  role=mongo
                  statefulset.kubernetes.io/pod-name=mongod-0
Annotations:      cni.projectcalico.org/containerID: e1e55e7b6468b60076ae2c76b344732142c8852838b5b354cd6bb06b09f21387
                  cni.projectcalico.org/podIP: 10.1.231.87/32
                  cni.projectcalico.org/podIPs: 10.1.231.87/32
Status:           Running
IP:               10.1.231.87
IPs:
  IP:           10.1.231.87
Controlled By:  StatefulSet/mongod
Containers:
  mongod-container:
    Container ID:  containerd://9c8594df7491036157b7958fdc77ca4e152dbfcf6f12a2ae5c6954bb8c63f637
    Image:         mongo
    Image ID:      docker.io/library/mongo@sha256:8bed0be3e86595283d67836e8d4f3f08916184ea6f2aac7440bda496083ab0c8
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      mongod
      --bind_ip
      0.0.0.0
      --replSet
      MainRepSet
    State:          Running
      Started:      Mon, 28 Nov 2022 13:54:56 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        200m
      memory:     200Mi
    Environment:  <none>
    Mounts:
      /data/db from mongo-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-75tsr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  mongo-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mongo-volume-mongod-0
    ReadOnly:   false
  kube-api-access-75tsr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

I have only one node in the cluster and everything I am doing is in 'default' namespace. Surprisingly I am able to connect to the mongodb statefulset from my host's terminal.

Answer 1

Did you try the hints from the troubleshooting page?

https://microk8s.io/docs/troubleshooting

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

   sudo iptables -P FORWARD ACCEPT
   sudo apt-get install iptables-persistent

or, if using ufw:

   sudo ufw default allow routed

The MicroK8s inspect command can be used to check the firewall configuration:

   microk8s inspect

A warning will be shown if the firewall is not forwarding traffic.