503 Service Unavailable - Master not discovered error in AKS

621 Views Asked by At

I have a AKS cluster (k1) with 3 nodes. I have deployed ECK operator using all_in_one.yaml. I have another AKS cluster(k2) which is running my application stack , i have installed filebeats in this cluster (k2) and it is shipping the logs to elasticsearch to another k8s cluster (k1). My k2 cluster configuration is 3 nodes (4 vCPU, 16 GiB RAM , 32GiB Temp Storage). My elasticsearch.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: dev-elasticsearch
spec:
  version: 7.9.2
  http:
    # Disable https
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        type: LoadBalancer
  nodeSets:
  - name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.store.allow_mmap: false
      xpack.security.enabled: true
      discovery.seed_hosts:
         - 10.x.x.x
         - 10.x.x.x
         - 10.x.x.x

Note : When i start the start es and kibana , everything works fine for about 30 mins and then i have service unavailable error in es {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Another thing i noticed is elastic-internal-init-filesystem is in Terminated state.

  1. Can someone help me resolve this issue ?
  2. Also how can i write the logs to a az storage with more space instead on default PV ?

Thanks for your help !!

0

There are 0 best solutions below