Guidance with Histogram data in Gauge and Heatmap

Grafana Prometheus
1

I have a histogram metric with 9 custom buckets I am collecting in an exporter I wrote. It basically logs the number of seconds between the start and end of a specific event. Values will fall between 0 and 2000 seconds.

These metrics are named:

iota_zmq_tx_confirm_time_bucket
iota_zmq_tx_confirm_time_count
iota_zmq_tx_confirm_time_sum

If I am using a Heatmap and I want to plot the frequency of buckets (not necessarily my predefined buckets) which metrics should I use and do I need any functions (delta() or rate()) to go along with them? Right now I use: delta(iota_zmq_tx_confirm_time_bucket[30m]) and the results appear to be close to what I would expect but I am very confused with how the delta is working for the Heatmap.

For the gauge with the Histogram mode enabled on the x-axis, I am using delta(iota_zmq_tx_confirm_time_bucket[30m])/60 to get a histogram that mostly matches what I see in the Heatmap but I suspect I am limiting my results by having the 30m in the delta? As in there are no results above 30 minutes and I am sure I should have some.

Would love to hear how other people are using these two items.

2

Nobody knows anything? Am I doing this wrong? Am I asking the question incorrectly?

3

Hi, @crholliday! I actually don’t understand what exactly a problem. Could you add some screenshots of your metrics? Delta works with the Heatmap in the same way as with other panels. The only difference of Heatmap mode is bucket values converting from cumulative histogram to regular:

http://docs.grafana.org/v5.1/features/datasources/prometheus/#query-editor

Heatmap format is suitable for displaying metrics having histogram type on Heatmap panel. Under the hood, it converts cumulative histogram to regular and sorts series by the bucket bound.

4

Hm, sorry, seems you use an old way to build a heatmap. Anyway, screenshots will be helpful, maybe a new mode for prometheus is suitable for your task.