- What Grafana version and what operating system are you using?
Grafana 12.3.0 in docker container
- What are you trying to achieve?
I want to plot the results of a nightly batch job.
- How are you trying to achieve it?
The batch job writes output to a json. A python script parses the file and sends the metrics to Mimir using prometheus remote write. For each metric, it sends a stale marker with a timestamp +15 seconds too. It sends the stale marker to prevent multiple dots in the graph when plotting in grafana.
- What happened?
When run the batch job and python script 3 times, at t=0m, t-5m and t=25m, Grafana plots the results and connects the dots with a line, but only when I select this specific timeframe. When I select last 2 days I don’t see the dots at all (but the legend is correct).
When I run the batch job nightly, and in Grafana select last 2 days, it also doesn’t show anything.
- What did you expect to happen?
Prometheus query to return all results and grafana to plot it.
- Can you copy/paste the configuration(s) that you are having problems with?
The promql query:
sum without(testid) (summary_response_time_ms{sut=“$SUT”, testenv=“$testenv”, testid=~“$testid”, tag=“$tag”})
During my analysis I ran this query in the above mentioned 25 minute interval:
sum without(testid) (summary_response_time_ms{tag=“myTransaction”, statistic=“max”})
Then I see three dots in the graph, but when using table view, I see at the first row the value, and then every 15 seconds (which is the default scrape interval) for 5 minutes NaN.
When I select last 24 hours, I only see two rows in the table with the value NaN. The batch job runs around 3AM, the two rows are at 3:02:00 and 3:04:00. It looks like Promql (or grafana) averages the numbers to a NaN result.
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
No
- Did you follow any online instructions? If so, what is the URL?
No