Grafana + Prometheus - combined query filtering: uri != "/page" AND method != "PUT"

Question

Our service receives both front-office(website) and back office calls. I want to separate these calls in my metrics in my grafana boards.

We know certain urls and methods combinations are only used by back office, like "PUT /page/draft" and "POST /page/publish". But there are several endpoints also used by front-office, like "GET /page/draft". So the query should combine two tags, like

http_server_requests({uri="/page/draft", method="PUT"}) OR ...

How can I use a variable of type query(or anything else) to filter them, so that in my graphs I can use http_server_requests({"$backoffice_requests"}) and http_server_requests({"$non_backoffice_requests"}) to draw two graphs?

Answer 1

So from what I see you have two options (maybe there are better ones out there but I can't think of any) IF doing the labeling directly in the instrumented service(s) is not an option.

Modifying the Prometheus config to add a label that identifies environments like "front" and "back" based on the values of other labels OR doing it at "query time" with Grafana template variables and PromQL. Both come with up- and downsides.

---

Option 1: Adding a label at scrape time

Advantages:

• Does not require changes in the clients. You probably use a small library that instruments your API and it does not support something like this I assume.
• Does not require doing stuff at runtime with label_replace or regex selectors overall nor using stuff specific to Grafana.
• Easy to implement and easy to use.
• Just another label you can use everywhere, not just Grafana. Meaning recording rules, alerts, the Prometheus web UI.

Disadvantages:

• I think the official standpoint of Prometheus people is that using the relabel config for this is a code smell. And when number of different label value combinations the regex increases as well. Though it will probably be just a very long (a|b|c) statement.
• Not possible to apply this label to series data in the past. If you add the label now and filter for it in Grafana, all the data before now will not be shown. Filtered out.
• Labels are hardcoded. If you change your opinion about what defines a "back office" series and you want to change the rule, the new rule will only be applied to data from now on. Again, you can't change the history.

How to do:

Add a metric_relabel_config (see here) to the respective job(s) in the Prometheus config. The basic idea is that you can regex match labels and add a new label with a specific value if a match occurs.

Here is an example:

scrape_configs: 
  - job_name: 'ecs'
    file_sd_configs:
      - files:
          - /discovery/tasks.json
    metric_relabel_configs:

      - action: replace
        source_labels: [__name__]
        target_label: scope
        replacement: unknown

      - action: replace
        source_labels: [uri, method]
        target_label: scope
        separator: ';'
        regex: '(PUT;\/page\/draft|POST\/whatever|and;so_on)'
        replacement: backoffice

      - action: replace
        source_labels: [uri, method]
        target_label: scope
        separator: ';'
        regex: '(GET;\/pagedraft)'
        replacement: backoffice

What this does is first adding the label scope with the default value unknown to all series and than continues to change the label based on the value of other labels.

---

Option 2: Separating at runtime in Grafana

tba