can configurable declarative Horizontal Pod AutoScaler (HPA) in seldon deployment (CRD) for multiple deployment environment is possible?

Question

Using CRD of SeldonDeployment I want to configure HPA following this example . The seldon deployment config is -

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - hpaSpec:
        maxReplicas: 3
        metrics:
        - resource:
            name: cpu
            targetAverageUtilization: 10
          type: Resource
        minReplicas: 1
      spec:
        containers:
        - image: seldonio/mock_classifier:1.5.0-dev
          imagePullPolicy: IfNotPresent
          name: classifier
          resources:
            requests:
              cpu: '0.5'
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      name: classifier
      type: MODEL
    name: example

When the above is applied this generates a deployment and a declarative HPA (along with other components). The deployment file is identical for QA, Alpha and Prod environments leading to the number of max / min replicas being the same. 1. Can the number of replicas be made configurable for different environments? i.e different values for max / min replicas for different envs (in the CRD SeldonDeployment config file).

Note: for other (non-Seldon) deployments, we use declarative HPA for each environment. But this is not possible in this case since the Seldon controller manager reverts any diff between the applied CRD and current (generated from the CRD) deployment.

2. More generalized question: how would we inject dynamic variables in CRD-based deployments?

Answer 1

I don't think that's a seldon-specific question.

I think you should define 3 charts like this for QA, Alpha and Prod. Each chart should have different values, and that's it!

Another thing is that this solution would duplicates the code - your static chart would be copied 3 times with just HPA params changed. One of the solutions for that is using helm charts. In this case, you write the chart template, that has a placeholder for the HPA values, and you just write 3 values files. That way you have one chart written and 3 values files that dynamically set the values for HPA.