Showing spikes at wide zooms

I’m using Grafana with a Prometheus datasource. One of the metrics I’m tracking is the rate of RPC errors in my services. Normally the rate is zero, but sometimes I’ll get brief spikes of dozens or hundreds of RPC errors in a minute.

However, I’m finding that the visibility of these spikes changes depending on the zoom level. If I’m zoomed to a day, I might see a spike, but zooming out to three days or seven days might make the spike disappear entirely, resulting in a line reading zero for the whole time period. The zeroed line is misleading, however, since there actually were errors during that time period.

Are there any recommendations or best practices for making sure that short, sharp spikes are visible at all zoom levels? Below is an example query I’ve been using:

sum by (grpc_service, grpc_method, grpc_code) (irate(grpc_server_handled_total{grpc_code="DeadlineExceeded"}[1m]))

if you use Grafana 4.2 I think you can use [$__interval] instead of [1m] this will change the interval to match the auto step parameter.

Using $__interval does not prevent spikes from disappearing. This is well visualized in the following issue.

Either the rate function has to include data points outside the given range or Grafana has to provide a value that leads to a slight oversampling. Björn initiated an issue that will solve this https://github.com/grafana/grafana/issues/21417

Just added this comment in case other people that are not that involved stumble upon it

1 Like