gradually warm java application in k8s with istio

Question

I have JVM-based applications that deliver excellent performance but need some time to “warm up” before reaching the top speed.

istio api (DestinationRule) does not support endpoint level weight, while envoy does.

I want a simple strategy: at the beginning when the pod is ready, the weight (qps) of that pod keeps a low level and increases over time

eg: lets I want the warm-up time to be 120 seconds. 2 pods have 50%-50% traffic, and when a new pod launches it will handle low traffic and increase over time and will reach 33% in 120 sec.

Let's take an example I have one deployment where 2 pods are running and I'm going to deploy a new version for my app when a new pod came traffic is equally divided by 33% each into 3 pods the new pods are unable to handle the traffic because the application needs some time to warm up and initiate the thread to handle the heavy load the same issue is coming during HPA (horizontal pod scaling).

I need a solution to warm up the pod and handle traffic gradually for new pods based on time duration

Answer 1

If you are using Kubernetes Ingress controller, then you can use the Peak EWMA load balancing mechanism.

You need add annotation for you ingress resource:

nginx.ingress.kubernetes.io/load-balance: ewma

See docs:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#load-balance

Answer 2

So you can do it in several ways, params in Destination Rules and traffic splitting by VirtualService

Simple way by using warmupDurationSecs:

The warmupDurationSecs parameter of LoadBalancerSettings section in Istio DestinationRule can be used to control the gradual increase of traffic to a new pod. When a new version of a deployment is applied, Istio can gradually increase the traffic to pods from updated deployment over a specified period of time to allow jvm compile all needed hotspot without throttling.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-service-destination-rule
spec:
  host: my-service
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
      warmupDurationSecs: 180s
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

So during 180 second istio will increase amount of traffic that will be send to pod, speed of growing will be almost linear:

More complex way with VirtualService:

Using Destination rule from previous code sample you can define a virtual service with two routes that have different weights:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

And then update it time to time to increase traffic that will be send to new version v2, see more in istio's tutorial.