Best Practice for Operators for how to get Deployment's configuration

Question

I am working on operator-sdk, in the controller, we often need to create a Deployment object, and Deployment resource has a lot of configuration items, such as environment variables or ports definition or others as following. I am wondering what is best way to get these values, I don't want to hard code them, for example, variable_a or variable_b.

Probably, you can put them in the CRD as spec, then pass them to Operator Controller; Or maybe you can put them in the configmap, then pass configmap name to Operator Controller, Operator Controller can access configmap to get them; Or maybe you can put in the template file, then in the Operator Controller, controller has to read that template file.

What is best way or best practice to deal with this situation? Thanks for sharing your ideas or points.

    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
            Labels:    ls,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image: "....",
                        Name: m.Name,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: port_a,
                            Name:          "tcpport",
                        }},
                        Env: []corev1.EnvVar{
                            {
                                Name:  "aaaa",
                                Value: variable_a,
                            },
                            {
                                Name:  "bbbb",
                                Value: variable_b,
                            },

Answer 1

Using enviroment variables

It can be convenient that your app gets your data as environment variables.

Environment variables from ConfigMap

For non-sensitive data, you can store your variables in a ConfigMap and then define container environment variables using the ConfigMap data.

Example from Kubernetes docs:

Create the ConfigMap first. File configmaps.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  special.how: very
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
  namespace: default
data:
  log_level: INFO

Create the ConfigMap:

kubectl create -f ./configmaps.yaml

Then define the environment variables in the Pod specification, pod-multiple-configmap-env-variable.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      env:
        - name: SPECIAL_LEVEL_KEY
          valueFrom:
            configMapKeyRef:
              name: special-config
              key: special.how
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: env-config
              key: log_level
  restartPolicy: Never

Create the Pod:

kubectl create -f ./pod-multiple-configmap-env-variable.yaml

Now in your controller you can read these environment variables SPECIAL_LEVEL_KEY (which will give you special.how value from special-config ConfigMap) and LOG_LEVEL (which will give you log_level value from env-config ConfigMap):

For example:

specialLevelKey := os.Getenv("SPECIAL_LEVEL_KEY")
logLevel := os.Getenv("LOG_LEVEL")

fmt.Println("SPECIAL_LEVEL_KEY:", specialLevelKey)
fmt.Println("LOG_LEVEL:", logLevel)

Environment variables from Secret

If your data is sensitive, you can store it in a Secret and then use the Secret as environment variables.

To create a Secret manually:

You'll first need to encode your strings using base64.

# encode username
$ echo -n 'admin' | base64
YWRtaW4=

# encode password
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm

Then create a Secret with the above data:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

Create a Secret with kubectl apply:

$ kubectl apply -f ./secret.yaml

Please notice that there are other ways to create a secret, pick one that works best for you:

• Creating a Secret using kubectl
• Creating a Secret from a generator
• Creating a Secret from files
• Creating a Secret from string literals

Now you can use this created Secret for environment variables.

To use a secret in an environment variable in a Pod:

1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
2. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in env[].valueFrom.secretKeyRef.
3. Modify your image and/or command line so that the program looks for values in the specified environment variables.

Here is a Pod example from Kubernetes docs that shows how to use a Secret for environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
  restartPolicy: Never

Finally, as stated in the docs:

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base64 decoded values of the secret data.

Now in your controller you can read these environment variables SECRET_USERNAME (which will give you username value from mysecret Secret) and SECRET_PASSWORD (which will give you password value from mysecret Secret):

For example:

username := os.Getenv("SECRET_USERNAME")
password := os.Getenv("SECRET_PASSWORD")

Using volumes

You can also mount both ConfigMap and Secret as a volume to you pods.

Populate a Volume with data stored in a ConfigMap:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        # Provide the name of the ConfigMap containing the files you want
        # to add to the container
        name: special-config
  restartPolicy: Never

Using Secrets as files from a Pod:

To consume a Secret in a volume in a Pod:

1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
2. Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].secret.secretName field equal to the name of the Secret object.
3. Add a .spec.containers[].volumeMounts[] to each container that needs the secret. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the secrets to appear. Modify your image or command line so that the program looks for files in that directory. Each key in the secret data map becomes the filename under mountPath.

An example of a Pod that mounts a Secret in a volume:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret