Metrics for Top-n Load, memory, CPU usage by host in Grafana

Question

We are trying to display the following in Grafana using the Bosun/OpenTSDB data source:

a. Hosts in descending order in terms of Top-n Load
b. Top 10 memory consuming processes
c. Top CPU usage consuming processes

However, we could not find suitable metrics for it.

How can this information be displayed?

Secondly, if the metrics are not available in Bosun/OpenTSDB, then how should you create or define new metrics for them?

Answer 1

Overview

1. Install the Bosun Grafana App plugin (Github Repo) and then setup the Bosun datasource.
2. Add a Table Panel, set your datasource to your new Bosun datasource.
3. Use limit(), sort(), and filter() functions as documented in Bosun's Expression Documentation

Table Example

For example, you could have an expression like the following for a table of top CPU:

$avg_cpu = avg(q("avg:$ds-avg:rate{counter,,1}:os.cpu{host=ny-*}{}", "$start", ""))
sort(limit(sort($avg_cpu, "desc"), 10), "desc")

note: sort is called twice so the table has a default sorting of by value

Graph Example

If you wanted to do a Graph panel instead of a table, you could use the filter():

$cpu = q("avg:$ds-avg:rate{counter,,1}:os.cpu{host=ny-*}{}", "$start", "")
$avg_cpu = avg(q("avg:$ds-avg:rate{counter,,1}:os.cpu{host=ny-*}{}", "$start", ""))
filter($cpu, limit(sort($avg_cpu, "desc"), 10))