Grafana table cannot show completed data sometimes

[haoyi13579]

Hi everyone,

I am working on using grafana to show some data in a table. However it sometimes will lack some data. It needs to refresh several times until it show all data correctly and after few refresh period, it will lack data again. I have checked the ES and Kibana, all data are completed restored. The data are recorded every 10 seconds and the ES read data every 30 seconds. On the grafana side I set the min time interval 1 minute.

the problem like this:

You can see that in the first picture the data are uncompleted which filled with “-”.(Please ignore the Chinese words)

I have discussed with my friends and it seems that the problem is caused by the grafana. When it read data from ES, it cannot read them completed.

The setting of the table is like this. I want to show the newest data in the table.

I have tried to find a solution but seems there is no one have this problem. Can you help me please?

THANKS!

[jangaraj]

Use Query Inspector (browser console) and check what is returned from ES. What is the total doc size returned by ES? How is your ES configured (index.refresh_interval, caching, …)?

[haoyi13579]

The total doc size sometimes will smaller than its expected value, just like it shows in the picture. It seems that the correct doc_count should be 9. The ES index refresh interval is 30s and I am sorry I cannot give the caching configuration now since my colleague set it.

[haoyi13579]

Hi jangaraj, I asked my colleague, the caching uses the default config.

[haoyi13579]

For your reference, I use two index in ES. The index format is like this.

Datetime, Hostname, Metric, Value, Value type (string or float), tag (to classify Value)

In these tow index, one value is date format and the other is float. One name is logstash-app-imds-metric-string-YYYY.MM.DD, the other is logstash-app-imds-metric-float-YYYY.MM.DD. So in the datasource configuration page I set the index name [logstash-app-imds-metric-*-]YYYY.MM.DD.

Hope this helps. Thanks again.

[jangaraj]

You can see that responses from ES are ok. But your config doesn’t look right. Are sure, that you can use ‘*’ in Grafana ES config?

Define multiply index names for one elasticsearch source

Hello everybody, I use searchguard to limit access to specific indexes. A user I created can for example see filebeat-* and metrics-. Currently I try to create a new source for Elasticsearch in Grafana, but when I use -YYYY.MM.DD I get an error message, because there are insufficient permissions for this user. It’s working when I use the specific index names (like [filebeat]-YYYY.MM.DD). Is there a way to create an array with names to set a bunch of names? I tried something like {filebeat-,…

[haoyi13579]

Thank you very much! I really appreciate your patience and help. I will check if this config is right.

I changed the now config and make it delayed for 30s. Everything runs smoothly. I still think this problem is something about grafana read the newest data from ES. At least, I solved this problem in some degree.

I would like to keep discussing this problem until it solved completely if someone wants to do that.

[rickadair]

I do see this same behavior with a faster update time and SingleStats. The data will not return and in looking at the Query Inspector is appears as if grafana is queuing up calls to ES or the proxy. Doesn’t matter which way it is configured. Once this queuing stops the data begins to display again.

[haileymoon2]

Hi, Im facing the same issue, can you share how you changed the config to delay for 30s?
Thanks alot.