Templated variable Multi-Value: Panels error when selecting more than 60

Grafana InfluxDB
1

Hi All,
I have a Dashboard with many panels, all with one or two templated queries.
My variable is defined through templating as a query to InfluxDB, it returns around 100 possible values now and the number will grow, and I need to be able to select multiple values.
Each panel shows a graph with “lines” for the selected values.
I have encountered a limit in the number of items I can select in the variable, exactly 60 items.
When I select 61 or more I start seeing errors in the panels (see screenshot-1 below), though if click view on any panel an view it alone they all seem to work, even selecting all values in the variable.
screenshot-1

Also, if I leave the the built-in annotation enabled I start receiving a pop-up red error message when I hit the 61 sites mark (see screenshot-2 below).
Again, this won’t appear if I view a single panel at a time.
screenshot-2

Can this be a bug caused by some sort of timeout in the JS when showing the full dashboard?

I’ve been looking everywhere hoping to find someone with the same issue but couldn’t find anything.

Grafana: 4.6.3-1
InfluxDB: 1.4.2-1

Thanks in advance,
Fran

2

PS: I’ve thought of a Javascript timeout because of what I’ve seen through Chrome’s Developer Tools (see screenshots below)

screenshot-3

screenshot-4

3

Are there any errors in the Grafana server logs on the backend?

4

Hi @daniellee,
Thanks for your reply.

The only messages I can see are lvl=info:

t=2018-02-22T11:23:29+0000 lvl=info msg="Request Completed" logger=context userId=9 orgId=1 uname=<...> method=GET path=/api/datasources/proxy/1/query status=502 remote_addr=<...> time_ms=24 size=0 referer="https://<...>/dashboard/db/fran-dashboard?refresh=15m&orgId=1&var-tla=ABS&var-tla=ACE&var-tla=CDT&var-tla=CIT&var-tla=DKD&var-tla=DKI&var-tla=EJF&var-tla=FNK&var-tla=IES&var-tla=MCL&var-tla=SAD&var-tla=YLO&var-tla=alpha&var-site=All"

5

The http status code 502 means bad gateway so the error is coming from your InfluxDB so it looks like you are ddosing your database with more queries than it can handle. Maybe you need to beef up your InfluxDB server or add clustering?

1 Like
6

Hi @daniellee,

It looks like you are right.

I was not seeing any errors in the InfluxDB log before.
The InfluxDB Data Source in our Grafana is set to be proxied via the Grafana backend.
When I changed the Access option to be Direct I started seeing an error in the InfluxDB log (when selecting the 61+ hosts).

So it looks like it’s indeed InfluxDB that can’t handle the number of queries combined with the big number of filters in the WHERE clause.
It’s a bit of a nasty error message too:
[panic:runtime error: invalid memory address or nil pointer dereference] goroutine 354995 [running]:
…

I’ll check with the InfluxDB guys and see what they say.

For the time being we changed our Dashboard to always SELECT by tag and not by hosts that have that tag.
Then we created a second Dashboard without tags, so the hosts can/have to be manually selected.
I doubt anyone will ever spend the time to select 61+ hosts there… :sweat_smile:

Thanks for your time Daniel, really appreciate it.

Fran

1 Like