Hi, I’m trying to generate statistics from log lines that have been ingested into Loki and we have successfully tested recording rules. The issue are that:
- recording rule queries are not aligned on a reasonable time boundary, like if my rule runs every 5min I want it to aggregate data from 00:00:00 to 00:04:59.999 then 00:05:00 to 00:09:59.999 and so on. now I am getting random snapshots of 5 min windows that don’t line up.
- if for some reason Loki is down or bogged up, all metrics will be missing data points
- I have not found a tool to backfill query metrics
Is there a separate tool / project that can help me query / backfill metrics from Loki and push them to Prom (well VictoriaMetrics but same same).As a comparison, VictoriaMetrics has a recording rule “replay” tool that does exactly this for recording rules, it makes perfectly step-aligned queries for each rule and can backfill the metrics. But of course it does not support Loki datasource & queries, just Prom. So I’m looking for something similar on Loki side. All ideas and horrible kludges welcome!