Subtract Micrometer/Prometheus different metrics for the same labels one from another

Question

I have 2 Micrometer

# HELP jms_payment_queue_in_total Current size of messages in JMS queue
# TYPE jms_payment_queue_in_total counter
jms_payment_queue_in_total{service_id="1056",} 2.0
jms_payment_queue_in_total{service_id="1050",} 4.0
jms_payment_queue_in_total{service_id="792",} 1.0
jms_payment_queue_in_total{service_id="584",} 1.0

and

# HELP jms_payment_queue_out_total Current size of messages in JMS queue
# TYPE jms_payment_queue_out_total counter
jms_payment_queue_out_total{success="success",service_id="1050",} 5.0
jms_payment_queue_out_total{success="success",service_id="977",} 22.0
jms_payment_queue_out_total{success="success",service_id="955",} 8.0

I like to plot a prometheus query to build jms_payment_queue_in_total{service_id="584",} - jms_payment_queue_out_total{success="success",service_id="584",} (for all possible same success and service_id fields). How to do that?

I tired to plot

abs_payment_total{service_id="1050"} - ignoring(success) jms_payment_queue_out_total{service_id="1050"}

in Grafana, where I definitely have points for both values, but it plots nothing. What is wrong here?

Answer 1

Just add missing filter on success:

abs_payment_total{service_id="1050",success=""} - ignoring(success)
abs_payment_total{service_id="1050",success="success"}

This query adds success="" filter to the left-hand side of - operator in order to filter out time series with non-empty success label.

The query adds success="success" filter to the right-hand side of - in order to select only time series with that label.

The query uses ignoring(success) in order to ignore success label when finding pairs of time series with identical labels on the left and the right side of - according to these docs.

P.S. you can use the following query in order to calculate the needed difference across all the service_id labels:

abs_payment_total{success=""} - ignoring(success) abs_payment_total{success="success"}

Answer 2

You can use promQL aggregation operators max(select maximum over dimensions)

max(jms_payment_queue_in_total{service_id="584"}) by (service_id)
- max(jms_payment_queue_out_total{success="success",service_id="584"}) by (service_id) 

or

max(max_over_time(jms_payment_queue_in_total{service_id="584"})[1s]) by (service_id)
- max(max_over_time(jms_payment_queue_out_total{success="success",service_id="584"})[1s]) by (service_id)