Creating sidecar Metricbeat with AWS EKS Fargate

788 Views Asked by At

I'm trying to create a deployment on AWS EKS with my application and metricbeat as sidecar, so I have the following YML:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-modules
  namespace: testframework
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: [ "https://${NODE_IP}:10250" ]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: "none"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: testframework
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    processors:
      - add_cloud_metadata:
      - add_tags:
          tags: ["EKSCORP_DEV"]
          target: "cluster_test"

    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false


    output.elasticsearch:
      index: "metricbeat-k8s-%{[agent.version]}-%{+yyyy.MM.dd}"
    setup.template.name: "metricbeat-k8s"
    setup.template.pattern: "metricbeat-k8s-*"
    setup.ilm.enabled: false
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testframework-initializr-deploy
  namespace: testframework
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testframework-initializr
  template:
    metadata:
      labels:
        app: testframework-initializr
      annotations:
        co.elastic.logs/enabled: 'true'
        co.elastic.logs/json.keys_under_root: 'true'
        co.elastic.logs/json.add_error_key: 'true'
        co.elastic.logs/json.message_key: 'message'
    spec:
      containers:
        - name: testframework-initializr
          image: XXXXX.dkr.ecr.us-east-1.amazonaws.com/testframework-initializr
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /health/liveness
              port: 8080
            initialDelaySeconds: 300
            periodSeconds: 10
            timeoutSeconds: 60
            failureThreshold: 5
          readinessProbe:
            httpGet:
              port: 8080
              path: /health
            initialDelaySeconds: 300
            periodSeconds: 10
            timeoutSeconds: 10
            failureThreshold: 3
        - name: metricbeat-sidecar
          image: docker.elastic.co/beats/metricbeat:7.12.0
          args: [
              "-c", "/etc/metricbeat.yml",
              "-e",
              "-system.hostfs=/hostfs"
          ]
          env:
            - name: ELASTIC_CLOUD_ID
              value: xxxx
            - name: ELASTIC_CLOUD_AUTH
              value: xxxx
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: config
              mountPath: /etc/metricbeat.yml
              readOnly: true
              subPath: metricbeat.yml
            - name: modules
              mountPath: /usr/share/metricbeat/modules.d
              readOnly: true
      volumes:
        - name: config
          configMap:
            defaultMode: 0640
            name: metricbeat-config
        - name: modules
          configMap:
            defaultMode: 0640
            name: metricbeat-modules
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prom-admin
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes"]
    verbs: ["get", "watch", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prom-rbac
subjects:
  - kind: ServiceAccount
    name: default
    namespace: testframework
roleRef:
  kind: ClusterRole
  name: prom-admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Service
metadata:
  name: testframework-initializr-service
  namespace: testframework
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: testframework-initializr
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: testframework-initializr-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - host: dev-initializr.test.net
      http:
        paths:
          - backend:
              serviceName: testframework-initializr-service
              servicePort: 80

Well, after startup the POD in AWS EKS, I got the following error in Kubernetes Metricbeat Container:

INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://IP_FROM_FARGATE_HERE:10250/stats/summary": dial tcp IP_FROM_FARGATE_HERE:10250: connect: connection refused

I tried to use the "NODE_NAME" instead "NODE_IP", but I got "No Such Host". Any idea how can I fix it?

0

There are 0 best solutions below