I am new to Graphite/Grafana and setting up some graphs for the first time. I probably have a basic question but unfortunately I am not able to find the right solution anywhere on the net.
I have a counter that shows incremental value - say ‘Total bytes transmitted’. This is a counter so it always shows the incremental value until it reaches its max limit.
I send this data roughly every hour to Graphite and would like to plot a graph showing only the delta between the data points. Then I want to setup an alert such that if the delta between the two provided values is say > 10000 then trigger an alert.
I have tried using derivatives, NonNegativeDerivates and also perSecond function but it does not appear to be doing the delta properly. I also tried keepLastValue and with that atleast I get some bars but the bar does not represent the correct data.
By default graphite assumes that there is a point every 60 seconds, so you’ll have a data point, then 59 NULLs, then another data point, which is why derivative etc won’t work properly. The default setting in Grafana is “null as connected” which essentially means that it ignores the nulls and gives you a graph of the intervals that have values.
The proper fix for this is to set up your storage schemas to match your data, then you won’t have all the nulls to contend with. In the meantime a band-aid solution would be to use alias(perSecond(summarize(com.$Hostname.TotDLBytes, "1hour", "last")), 'Total DL Bytes') which will effectively take the highest counter value in each hour and then plot the per-second rate based off those.
Thank you Dan for your help and quickly identifying the issue. Earlier from reading the Graphite doc I was getting an impression that storage schema were just for data retention and purging but now I understand how these settings can distort my calculations and graphs.
