I am working on using grafana to show some data in a table. However it sometimes will lack some data. It needs to refresh several times until it show all data correctly and after few refresh period, it will lack data again. I have checked the ES and Kibana, all data are completed restored. The data are recorded every 10 seconds and the ES read data every 30 seconds. On the grafana side I set the min time interval 1 minute.
Use Query Inspector (browser console) and check what is returned from ES. What is the total doc size returned by ES? How is your ES configured (index.refresh_interval, caching, …)?
The total doc size sometimes will smaller than its expected value, just like it shows in the picture. It seems that the correct doc_count should be 9. The ES index refresh interval is 30s and I am sorry I cannot give the caching configuration now since my colleague set it.
For your reference, I use two index in ES. The index format is like this.
Datetime, Hostname, Metric, Value, Value type (string or float), tag (to classify Value)
In these tow index, one value is date format and the other is float. One name is logstash-app-imds-metric-string-YYYY.MM.DD, the other is logstash-app-imds-metric-float-YYYY.MM.DD. So in the datasource configuration page I set the index name [logstash-app-imds-metric-*-]YYYY.MM.DD.
Thank you very much! I really appreciate your patience and help. I will check if this config is right.
I changed the now config and make it delayed for 30s. Everything runs smoothly. I still think this problem is something about grafana read the newest data from ES. At least, I solved this problem in some degree.
I would like to keep discussing this problem until it solved completely if someone wants to do that.
I do see this same behavior with a faster update time and SingleStats. The data will not return and in looking at the Query Inspector is appears as if grafana is queuing up calls to ES or the proxy. Doesn’t matter which way it is configured. Once this queuing stops the data begins to display again.
