Can't access my local kubernetes service over the internet

Question

Implementation Goal

Expose Zookeeper instance, running on kubernetes, to the internet.

(configuration & version information provided at the bottom)

Implementation Attempt

I currently have a minikube cluster running on ubuntu 14.04, backed by docker containers. I'm running a bare metal k8s cluster, and I'm trrying to expose a zookeeper service to the internet. Seeing as my cluster is not running on a cloud provider, I set up metallb, in order to provide a network-loadbalancer implementation for my zookeeper service.

On startup everything looks good, an external IP is assigned and I can access it from the same host via a curl command.

$ kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-5c9894b5cd-9gh8m   1/1     Running   0          5h59m
speaker-j2z8q                 1/1     Running   0          5h59m


$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
kubernetes   ClusterIP      10.xxx.xxx.xxx  <none>          443/TCP                         6d19h
zk-cs        LoadBalancer   10.xxx.xxx.xxx  172.1.1.x       2181:30035/TCP                  56m
zk-hs        LoadBalancer   10.xxx.xxx.xxx  172.1.1.x       2888:30664/TCP,3888:31113/TCP   6m15s

When I curl the above mentioned external IP's, I get a valid response

$ curl -D- "http://172.1.1.x:2181"
curl: (52) Empty reply from server

So far it all looks good, I can access the LB from outside the cluster with no issues, but this is where my lack of Kubernetes/Networking knowledge gets me.I'm finding it impossible to expose this LB to the internet. I've tried running minikube tunnel which I had high hopes for, only to be deeply disappointed.

Running a curl command from another node, whilst minikube tunnel is running will just see the request time out.

$ curl -D- "http://172.1.1.x:2181"
curl: (28) Failed to connect to 172.1.1.x port 2181: Timed out

At this point, as I mentioned before, I'm stuck. Is there any way that I can get this service exposed to the internet without giving my soul to AWS or GCP?

Any help will be greatly appreciated.

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  selector:
    app: zk
  ports:
    - port: 2888
      targetPort: 2888
      name: server
      protocol: TCP
    - port: 3888
      targetPort: 3888
      name: leader-election
      protocol: TCP
  clusterIP: ""
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  selector:
    app: zk
  ports:
    - name: client
      protocol: TCP
      port: 2181
      targetPort: 2181
  type: LoadBalancer
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: zookeeper
          imagePullPolicy: Always
          image: "library/zookeeper:3.6"
          resources:
            requests:
              memory: "1Gi"
              cpu: "0.5"
          ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: server
            - containerPort: 3888
              name: leader-election
          volumeMounts:
            - name: datadir
              mountPath: /var/lib/zookeeper
            - name: zoo-config
              mountPath: /conf
      volumes:
        - name: zoo-config
          configMap:
            name: zoo-config
      securityContext:
        fsGroup: 2000
        runAsUser: 1000
        runAsNonRoot: true
  volumeClaimTemplates:
    - metadata:
        name: datadir
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zoo-config
  namespace: default
data:
  zoo.cfg: |
    tickTime=10000
    dataDir=/var/lib/zookeeper
    clientPort=2181
    initLimit=10
    syncLimit=4

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.1.1.1-172.1.1.10

minikube: v1.13.1
docker: 18.06.3-ce

Answer 1

You can do it with minikube, but the idea of minikube is just to test stuff on your local environment. So, by default, it does not have the correct IPTable permissions, and yes you can adjust that, but if your goal is only to use without any loud provider, I'll higly recommend you to use kubeadm (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/).

This tool will provide you a very customizable cluster configuration and you will be able to set your network problems without headaches.