Yes … Almost …
Context: We can’t have Grafana connecting directly to our production instances for reporting (data access is restricted as the DB contains PII and we’re not supposed to query the DB directly from where our Grafana is), but we can have aggregated data getting out.
We’re prototyping a few things and one of the simplest options so far is to get the reporting data cubes alongside the monitoring metrics we’re already gathering.
I’m prototyping using prometheus to gather these. My prototype metrics look like:
test_data_metric{date="2023-12-01",facility="facility A"} 5
test_data_metric{date="2023-12-01",facility="facility B"} 12
test_data_metric{date="2023-12-01",facility="facility C"} 200
test_data_metric{date="2024-01-01",facility="facility A"} 12
test_data_metric{date="2024-01-01",facility="facility B"} 50
test_data_metric{date="2024-01-01",facility="facility C"} 230
test_data_metric{date="2024-02-01",facility="facility A"} 24
test_data_metric{date="2024-02-01",facility="facility B"} 100
test_data_metric{date="2024-02-01",facility="facility C"} 340
test_data_metric{date="2024-03-01",facility="facility A"} 48
test_data_metric{date="2024-03-01",facility="facility B"} 75
test_data_metric{date="2024-03-01",facility="facility C"} 360
test_data_metric{date="2024-04-01",facility="facility A"} 100
test_data_metric{date="2024-04-01",facility="facility B"} 150
test_data_metric{date="2024-04-01",facility="facility C"} 380
test_data_metric{date="2024-05-01",facility="facility A"} 200
test_data_metric{date="2024-05-01",facility="facility B"} 120
test_data_metric{date="2024-05-01",facility="facility C"} 400
Once they are ingested, I take them as “Instant” to avoid duplicates, then apply a few transformations:
- Change “date” type to timestamp
- Rename “date” as “Time”
- Partition by Values on “facility”,
I’ve got something that looks EXACTLY like what I want: one line per “facility”, the dates being what I got in my “date” label. Wow. I was not expecting it to be that easy.
I was super excited, because that would solve our problem with ZERO development, just some configuration on existing tools.
Then I tried to zoom in and the data is gone. Of course it is.
I fully understand why I have this behaviour. When i zoom on a past time range, Grafana ties to fetch the data as it was inside that time range, and my metrics were not existant at the time (even if they were I would likely prefer to take the current ones as well).
Is there a way to tell grafana to use the current value for a graph, even if that means taking value outside of the selected time range ?