Horizontal Pod Autoscaling Targets shows <unknown>/50% with "kubectl get hpa"

Question

Attempting to deploy autoscaling to my cluster, but the target shows "unknown", I have tried different metrics servers to no avail. I followed [this githhub issue](https"//github.com/kubernetes/minikube/issues4456/) even thought I'm using Kubeadm not minikube and it did not change the problem.

I also followed this Stack post with no success either.

I'm running Ubuntu 20.0.4 LTS.

Using kubernetes version 1.23.5, for kubeadm ,kubcectl, ect

Following the advice the other stack post, I grabbed the latest version via curl

curl -L https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

I edited the file to be as followed:

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubectl-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent

I then ran kubectl apply -f components.yaml

Still did not work:

$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE teastore-webui-hpa Deployment/teastore-webui <unknown>/50% 1 20 1 20h

Another suggestion was specifically declaring limits.

$ kubectl autoscale deployment teastore-webui --max=20 --cpu-percent=50 --min=1
horizontalpodautoscaler.autoscaling/teastore-webui autoscaled
 
group8@group8:~/Downloads/TeaStore-master/examples/kubernetes$ kubectl get hpa
NAME                 REFERENCE                   TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
teastore-webui       Deployment/teastore-webui   <unknown>/50%   1         20        0          4s
teastore-webui-hpa   Deployment/teastore-webui   <unknown>/50%   1         20        1          20h

That also did not work.

Here is an exert of the deployment and service config that I'm trying to autoscale.

    spec:
      containers:
        - name: teastore-webui
          image: descartesresearch/teastore-webui
          ports:
            - containerPort: 8080
          env:
            - name: HOST_NAME
              value: "teastore-webui"
            - name: REGISTRY_HOST
              value: "teastore-registry"
          resources:
            requests:
              cpu: "250m"
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: teastore-webui-hpa
  labels:
    app: teastore
spec:
  maxReplicas: 20
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: teastore-webui
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
---
apiVersion: v1
kind: Service
metadata:
  name: teastore-webui
  labels:
    app: teastore
    run: teastore-webui
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30080
      protocol: TCP
  selector:
    run: teastore-webui

Based on other suggestions I have the resource specifically declared as cpu with 50% utilization, and CPU requests are set to 250 milicores.

   $kubectl describe hpa
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                  teastore-webui
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 02 Apr 2022 16:07:25 -0400
Reference:                                             Deployment/teastore-webui
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          20
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age                    From                       Message
  ----     ------                        ----                   ----                       -------
  Warning  FailedComputeMetricsReplicas  29m (x12 over 32m)     horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  Warning  FailedGetResourceMetric       2m12s (x121 over 32m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Answer 1

Syntaxerror on line 6 of this yaml. It needs to be - --kubelet-insecure-tls and not - --kubectl-insecure-tls

spec:
  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubectl-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-use-node-status-port
    - --metric-resolution=15s
    image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
    imagePullPolicy: IfNotPresent

Noticed by checking the log files with

kubectl logs -f metric-server -n kube-system

Thank you David Maze for the comment.