I have an existing query that shows plenty of data:
Looking in elasticsearch, we can see that each result has a value in the
But when I try to group my Grafana query by
proxy_upstream_name, all the data in my chart disappears.
I expected it to appear similar to the “Expected” image above, but with a line and legend entry for “mad-pig-apicontent-80” and all other
proxy_upstream_name values .
What am I doing wrong here?
Can you investigate the query sent to Elasticsearch and elasticsearch response? Chrome dev tools network tab. Maybe there is something wrong with the query or the ES response
How to troubleshoot metric queries:
If you get a panel query error or not seeing the data you expect to see the most important thing to do
is view the query Grafana sends to your database and what the raw response is. This information is not built into Grafana yet but can be easily investigated using Browser Dev Tools.
To open Chrome Dev Tools (F12, Ctrl+Shift+I on windows, Cmd+Opt+I on Mac), go the network tab. Hit refresh on the Grafana dashboard so Grafana sends a new data query. Try to find the query in the list of network c…
Thanks for the pointers! Elasticsearch response showed the following error message:
Fielddata is disabled on text fields by default. Set fielddata=true on [proxy_upstream_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
Which lead me to
this Elasticsearch page and then I queried my
logstash-* mappings in elasticsearch like so:
Looked at the mapping for
Saw that it already had a
keyword, so I merely had to change my group by term name to
proxy_upstream_name.keyword and groupings worked! Hooray.