Graph panel randomly stopped working

Hi There,

Am facing a very odd issue where I have had the dashboard built and working with no-issues all these days but randomly couple days ago, one of the graph panel is showing disk utilization as 0% even though that is not the case. Am looking at data inside ElasticSearch and it is reporting right numbers but some how Grafana doesnt show the same. Can someone let me know of any known issues that could possible cause this?

Grafana: 4.5.2
ElasticSearch: 6.1.1
metricbeat: 6.1.1

Thanks,

Vish

Hi,

Can you please include a json document that is like 10 minutes after the time of graph series showing zero? If you change time ranges do you see any differences?

Also, you could use the query inspector feature to check the response that is actually coming back from the query before rendered in graph.

Also, I think the following topic sounds familiar to your problem: Data Missing Past Midnight (ES Data Source)

Marcus

Thank you for the quick reponse @mefraimsson and also for sharing that thread. So am seeing the same thing as being reported in other thread. Upon further checking things, based off my learning from that thread, am also not able to do visualization for that very same metric in Kibana either and also have noticed below error in LogStash for a couple metrics off which one is system.filesystem.used.pct.

Am assuming this has something to do with Elastic stack and something to do with that field not being indexed properly. Am going to further look into this one, possibly with some help from ELK community and can update this thread with my findings if that would be of any help.

Could not index event to Elasticsearch. {:status=>400, :action=>[“index”, {:_id=>nil, :_index=>“metricbeat-2018.02.21”, :_type=>“doc”, :_routing=>nil}, #LogStash::Event:0x5e26bbbc], :response=>{“index”=>{"_index"=>“metricbeat-2018.02.21”, “_type”=>“doc”, “_id”=>“Ei6otWEBR01LZFcC2KTi”, “status”=>400, “error”=>{“type”=>“illegal_argument_exception”, “reason”=>“mapper [system.diskio.iostat.request.avg_size] cannot be changed from type [float] to [long]”}}}}

1 Like

This is what I was doing wrong. After updating with right index name in logstash, issue has resolved.

1 Like

HI @s405121 Can you please suggest me to create the disk space graph as yours?

What are the parameter i need to include to create the graph against GB as Y-AXIS?