I have a histogram metric with 9 custom buckets I am collecting in an exporter I wrote. It basically logs the number of seconds between the start and end of a specific event. Values will fall between 0 and 2000 seconds.
These metrics are named:
If I am using a Heatmap and I want to plot the frequency of buckets (not necessarily my predefined buckets) which metrics should I use and do I need any functions (delta() or rate()) to go along with them? Right now I use:
delta(iota_zmq_tx_confirm_time_bucket[30m]) and the results appear to be close to what I would expect but I am very confused with how the delta is working for the Heatmap.
For the gauge with the Histogram mode enabled on the x-axis, I am using
delta(iota_zmq_tx_confirm_time_bucket[30m])/60 to get a histogram that mostly matches what I see in the Heatmap but I suspect I am limiting my results by having the 30m in the delta? As in there are no results above 30 minutes and I am sure I should have some.
Would love to hear how other people are using these two items.