How to customize ConfigMap for Container insights using Azure Monitor Agent on AKS?

Question

To collect logs from the AKS container application insights can be enabled. This automatically creates a deployment called ama-logs-rs This deployment has an associated ConfigMap: ama-logs-rs-config

The Terraform resource for deploying the cluster and node pool does not have any options to configure any settings of the monitoring agent.

The suggestion from Microsoft seems to be to edit it manually, as described here

However, that does not ensure that for every node scaling event these settings are automatically transferred, which means that either you constantly have to manually check and update the ConfigMap, or write some custom job that does this periodically or based on some triggers.

Is there a better way to customize specific settings for the ama on AKS?

Answer 1

A custom config map is the way to go, but using the correct name is important.

The deployment that controls the ama pods has some default optional mounts, one of them is the config map "container-azm-ms-agentconfig"

By default the configmap does not exist, but if you create it, it is automatically picked up.

This is the very minimal configmap we used in the end:

apiVersion: v1
data:
    log-data-collection-settings: |-
        # Log data collection settings
        #     # Any errors related to config map settings can be found in the KubeMonAgentEvents table in the Log Analytics workspace that the cluster is sending data to.
        #         # For additional setting see: https://raw.githubusercontent.com/microsoft/Docker-Provider/ci_prod/kubernetes/container-azm-ms-agentconfig.yaml

        [log_collection_settings]
          [log_collection_settings.env_var]
            # In the absense of this configmap, default value for enabled is true
            enabled = false
kind: ConfigMap
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system

kubectl describe deploy ama-logs-rs

OUT

Volumes:
...
    settings-vol-config:
        Type:      ConfigMap (a volume populated by a ConfigMap)
        Name:      container-azm-ms-agentconfig
        Optional:  true

Answer 2

To customize ConfigMap for Container Insights using Azure Monitor Agent on AKS, you can follow the steps below:

1. Download the ConfigMap file ama-logs-rs-config and rename it to configmap-ama.yaml. (Instead of modifying the auto-generated ama-logs-rs-config, create your own ConfigMap with a different name. This prevents your custom configurations from being overwritten by system updates.)

2. Modify the configmap-ama.yaml file to include the specific settings you want to customize.

3. Apply the modified ConfigMap to your AKS cluster using the kubectl apply command.

For example, to enable the collection of kube-audit logs, you can modify the configmap-ama.yaml file as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ama-logs-rs-config
  namespace: kube-system
data:
  config.yaml: |-
    sources:
      - tail:
          path: /var/log/kube-audit/audit.log
          parser: json
          exclude_paths:
            - /var/log/kube-audit/audit.log.*
    sinks:
      - azuremonitorlogs:
          customer_id: <your-customer-id>
          shared_key: <your-shared-key>
          log_type: ContainerLog
          log_level: Informational
          metadata:
            cluster_name: <your-cluster-name>
            container_type: <your-container-type>
            container_name: <your-container-name>

Then, apply the modified ConfigMap to your AKS cluster using the following command:

kubectl apply -f configmap-ama.yaml

Note that you can also use Azure Policy to ensure consistent configuration across multiple clusters.

However, as I explained above, if you want to ensure that configuration changes persist across node scaling events for the Azure Monitor Agent (AMA) on Azure Kubernetes Service (AKS), you can't rely on the default ConfigMap because it might be overridden by the system.

Alternative ways are-

Use AKS Start-up Script, AKS allows you to run a start-up script (/etc/kubernetes/init.d/) on the AKS nodes. You could add a script that applies the custom configuration to the ama-logs-rs-config ConfigMap after the node has initialized.

Or

use an init container in your pod specifications that configures the AMA settings before the main containers start. Each of these methods has its pros and cons, and the best approach depends on your operational preferences.

Remember, directly editing auto-generated ConfigMaps like ama-logs-rs-config is not recommended as changes may be lost during updates or scaling operations. Creating a separate, custom ConfigMap with your settings provides a more reliable solution.

References:

• Monitor Kubernetes clusters using Azure services and cloud native tools
• Collect Prometheus metrics with Container insights
• Monitor Azure Kubernetes Service (AKS) control plane metrics (preview)