Metric frequently falls to zero

Grafana Elasticsearch
1

we found the data on the graph fall to zero frequently at last minute, refer to below image.

Screenshot_2017-10-24_21-07-01
Screenshot_2017-10-24_21-07-013160×1730 406 KB

we basically use two kinds of metric collecting:

1.collectd collect devices snmp, and centralize them to logstash, then to elasticsearch. (interval: 30s)

2.logstash http_poller plugin to collect several of our apis metric. (interval: 60s)

both of the graph have the same thing happen, even in single stat panel (show 0).

we build this for out OPS team to replace PRTG for monitor near real time statistic, we’ll know something went wrong once the metric falls to zero.

first we thought, maybe the X-Axis just go beyond the data collection, so the data just need some time to get collected (logstash/colectd not yet trigger), fair enough.

since collect metric takes time, so it’s okey with us to add some timeshift (eg. now-1h to now-1m) to avoid this confusing, frequently happening falling graph.

BUT, IT JUST HAPPENENED, please refer to the image, such time point (21:04) should have value, doesn’t it?

Screenshot_2017-10-24_21-07-42
Screenshot_2017-10-24_21-07-423160×1730 404 KB

we are greatly wondering what is going on and how can we avoid that?
let us know if we can provide other helpful information, msearch request/response, raw data…
many thanks.

2

This isn’t a Grafana feature for this. Data sources like InfluxDB and Graphite have functions for this (fill for InfluxDB and keepLastValue for Graphite). Maybe someone else knows how to do this with Elasticsearch?

3

Thanks, I’ve seen another post of this is using Graphite as data source, and with your suggestion is use keepLastValue.
but unfortunately we are using Elasticsearch, could you suggest any work around or where can we find the answer for?

another thing that’s pretty weird is that even we add time shift, the graph shows zero which it shouldn’t.
as you can see in the first pic, it’s 21:06 that don’t have value yet, 21:04 already have.

at second pic, I take the screenshot at 21:07, and the time range selection is now-30m to now-3m,
and turns out the graph showing 21:04 become to zero.

are we misunderstanding the time range selection?