I have a system that imports records into a database and then logs the import duration as hh:mm:ss.fffffff. These logs are imported into Loki, and I'm trying to create a graph of the import times in Grafana. Below is a sample log line:
2021-10-13T20:59:41.546540906Z stdout F [10] INFO Import.Actions.Base.ImportController - Import time 00:00:04.3612494
I have the query below, which uses a regex to parse out the minutes and seconds and then uses unwrap and avg_over_time to graph those values, but it only works for times below 1 minute:
sum(avg_over_time({app="import"} | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | processing_secs > 0 | unwrap processing_secs [2m]))
When the duration is something like 00:01:05.000000, then the time gets misreported as 5s, not 1m 5s. Is there a way to either get Loki / Grafana to understand 01:05 as a duration OR to do math on the two values I extracted and graph that? I tried to do something like:
sum(avg_over_time({app="import"} | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | processing_secs > 0 | unwrap processing_mins * 60 + processing_secs [2m]))
but that doesn't work.
I also tried the following to create a duration (the docs indicate Loki understands duration https://grafana.com/docs/loki/latest/logql/), but I got an error with this as well:
avg_over_time({app="import"} |= "Import time" | regexp `Import time 00:(?P<processing_mins>\d\d):(?P<processing_secs>\d+\.\d+)` | label_format duration="{{.processing_mins}}m{{.processing_secs}}s" | unwrap duration[1m])
Gave me this error:
pipeline error: 'SampleExtractionErr' for series: '{error="SampleExtractionErr", app="import", container="eps-aapimport", duration="00m04.0713535s", filename="/var/log/pods/import-fb874d5fc-9v64l_522d5b82-78c0-4780-a256-f3b95b62e269/import/0.log", job="stable/import", namespace="stable", pod="import-fb874d5fc-9v64l", pod_template_hash="fb874d5fc", processing_mins="00", processing_secs="04.0713535"}'. Use a label filter to intentionally skip this error. (e.g | error!="SampleExtractionErr"). To skip all potential errors you can match empty errors.(e.g error="") The label filter can also be specified after unwrap. (e.g | unwrap latency | error="" )