How do you extract and manipulate multiple values from log lines via LogQL for graphing?

2.3k Views Asked by At

I have a system that imports records into a database and then logs the import duration as hh:mm:ss.fffffff. These logs are imported into Loki, and I'm trying to create a graph of the import times in Grafana. Below is a sample log line:

2021-10-13T20:59:41.546540906Z stdout F [10] INFO Import.Actions.Base.ImportController - Import time 00:00:04.3612494

I have the query below, which uses a regex to parse out the minutes and seconds and then uses unwrap and avg_over_time to graph those values, but it only works for times below 1 minute:

sum(avg_over_time({app="import"} | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | processing_secs > 0 | unwrap processing_secs [2m]))

When the duration is something like 00:01:05.000000, then the time gets misreported as 5s, not 1m 5s. Is there a way to either get Loki / Grafana to understand 01:05 as a duration OR to do math on the two values I extracted and graph that? I tried to do something like:

sum(avg_over_time({app="import"} | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | processing_secs > 0 | unwrap processing_mins * 60 + processing_secs [2m]))

but that doesn't work.

I also tried the following to create a duration (the docs indicate Loki understands duration https://grafana.com/docs/loki/latest/logql/), but I got an error with this as well:

avg_over_time({app="import"} |= "Import time" | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | label_format duration="{{.processing_mins}}m{{.processing_secs}}s" | unwrap duration[1m])

Gave me this error:

pipeline error: 'SampleExtractionErr' for series: '{error="SampleExtractionErr", app="import", container="eps-aapimport", duration="00m04.0713535s", filename="/var/log/pods/import-fb874d5fc-9v64l_522d5b82-78c0-4780-a256-f3b95b62e269/import/0.log", job="stable/import", namespace="stable", pod="import-fb874d5fc-9v64l", pod_template_hash="fb874d5fc", processing_mins="00", processing_secs="04.0713535"}'. Use a label filter to intentionally skip this error. (e.g | error!="SampleExtractionErr"). To skip all potential errors you can match empty errors.(e.g error="") The label filter can also be specified after unwrap. (e.g | unwrap latency | error="" )

0

There are 0 best solutions below