How to calculate CPU usage in Cadvisor to be equal to docker stats

Question

I have a cpu usage data in Cadvisor (12 cores) . But I want to calculate to be equal to docker stats.

First time I use this query but it's not work because the value in Prometheus not be the same with docker stats and I want the value to percentage.

rate(container_cpu_usage_seconds_total{name="XXXX"}[1m])

What should I do to calculate?

Answer 1

docker stats shows instant CPU usage at the current time, while cadvisor exports the total CPU time used by the particular container via container_cpu_usage_seconds_total metric. Such metric type is called counter. Prometheus scrapes cadvisor metrics with the configured interval, which is called scrape_interval. It can be configured in prometheus.yaml file at global section or individually per each scrape_config section. By default it is set to one minute.

This means that Prometheus doesn't collect instant CPU usage for each container from cadvisor. Instead, it collects the total per-container CPU usage every scrape_interval. So it can calculate only an average CPU usage over the scrape_interval or over any interval bigger than the scrape_interval. The rate function must be used for calculating the average CPU usage over the given interval specified in square brackets. For example, rate(container_cpu_usage_seconds_total[5m]) returns the average per-container CPU usage over the last 5 minutes.

P.S. Note that irate function doesn't return an instant CPU usage - it returns the average CPU usage over the last two scrapes. For example, if the scrape_interval equals to 30 seconds, then irate(container_cpu_usage_seconds_total[5m]) returns average CPU usage over the last 30 seconds. See also this article.