We have a bunch of custom servers running running a custom service that outputs stats about its operation in a CSV format of about 12 columns and a variable number of lines with a header row and column over HTTPS (https://servername:port/my/stats/url/csv. Currently I scrape around 30 of these services using Python and Pandas to pull the data into a dataframe, clean it up and massage it, and then output it as JSON to a log file that we then read in and do reports on. Each cell in the frame is an ever-increasing counter starting at 0 from the last time it was restarted. These stats change constantly and are updated about once every second.
I want to get this into grafana either using the JSON-formatted logs or the CSV https endpoint so I can store it long-term and query it over time. I know Loki can pull the logs in, but I don't really want them as logs, the entries in the file contain data and a count of different log entries that have a specific keyword is neither interesting nor informative. The JSON entries in the log file take the form of, and there are a few lines written every 15 minutes.
{"timestamp":1700512202,"service_name":"my-service-name","stats-type":"user-stats","user":"myusername","secondsSinceseen":263168,"responses":323,"auth_errors":24,"line_errors":0,"encode_errors":0,"internal_error":0}
and the csv takes the form:
stats-type,user-stats,secondsSinceSeen,responses,auth,yadda,yadda
gwuser,myUserName,959267,66359,0,0,0
I need the data from the columns to be transformed into queryable metrics in Grafana/Prometheus. I thought the easiest way might be to install Graphite or Carbon, but there is a ton of overhead that seems to accompany them and I'd really rather not. What's going to be the best/easiest way to get these metrics into Grafana/Prometheus?
I'm running prometheus/Grafana and have tried using Loki/Promtail to ingest the JSON logs but the ability to derive stats from this seems limited to counting the number of lines with specific text in it. My log file contains the data I need, I don't need to generate meta data on what's contained. I need to either extract the fields from the file entries into metrics or see if I can scrape the CSV pages over HTTP directly, which doesn't seem to be the case.
The CSV Datasource plugin does not appear to convert and store these metrics for future reporting or visualization, it seems to be a point-in-time datasource, which wouldn't work for my application. This seems to be true for the JSON datasource plugin as well.