InfluxDB clause "where not like / where not starting with" is causing Network Error: undefined(undefined)

Grafana Time Series Panel
1

Hi Guys,

The following InfluxDB query used to work (I think in Grafana v5.x versions), but just noticed it is now causing an error “Network Error: undefined(undefined)” is shown in the panel top right corner). The query doesn’t return any data and shows that error immediately on clicking Query Inspector:

image
image1145×367 31 KB

SELECT non_negative_derivative(last(“ifHCInOctets”), 1s) AS “In”, non_negative_derivative(last(“ifHCOutOctets”), 1s) *-1 AS “Out (-)” FROM “snmp.hosts.ifXTable” WHERE (“agent_host” =~ /^$host$/ AND “ifName” !~ /^docker|veth/) AND $timeFilter GROUP BY time($__interval), “agent_host”, “ifName” fill(null)

The error itself appears to be due to the “ifName” !~ /^docker|veth/ (i.e. where ifName does not start with “docker” or “veth”) clause so just wondered whether someone can point me in the direction to fix this SQL to get it working with current Grafana?

I’m running Grafana 6.1.4 and InfluxDB

Thanks!

2

In the linked question, I asked if they could examine the response and error in the Chrome Dev Tools. Could you do the same so that we can get more information about the error:

image
image1281×1324 169 KB

3

Hi @daniellee,

The attached is the output from Chrome Dev Tools for that query when I click on “Query Inspector”. The same error is presented by simply refreshing the dashboard and waiting for the error to occur on that panel…

VM94:1 GET https://<GRAFANA_URL>:<GRAFANA_PORT>/api/datasources/proxy/1/query?db=telegraf&q=SELECT%20non_negative_derivative(last(%22ifHCInOctets%22)%2C%201s)%20AS%20%22In%22%2C%20non_negative_derivative(last(%22ifHCOutOctets%22)%2C%201s)%20*-1%20AS%20%22Out%20(-)%22%20FROM%20%22snmp.hosts.ifXTable%22%20WHERE%20(%22agent_host%22%20%3D~%20%2F%5E192%5C.168%5C.1%5C.254%24%2F%20AND%20%22ifName%22%20!~%20%2F%5Edocker%7Cveth%2F)%20AND%20time%20%3E%3D%201556812800000ms%20GROUP%20BY%20time(10m)%2C%20%22agent_host%22%2C%20%22ifName%22%20fill(null)&epoch=ms net::ERR_SPDY_PROTOCOL_ERROR 200
(anonymous) @ VM94:1
(anonymous) @ angular.js:12759
(anonymous) @ angular.js:12492
(anonymous) @ angular.js:12244
(anonymous) @ angular.js:17051
(anonymous) @ angular.js:17095
$digest @ angular.js:18232
$apply @ angular.js:18531
(anonymous) @ angular.js:18832
h @ angular.js:6362
(anonymous) @ angular.js:6642
setTimeout (async)
i.defer @ angular.js:6640
$applyAsync @ angular.js:18831
(anonymous) @ angular.js:12544
w @ angular.js:12785
y.onload @ angular.js:12702
load (async)
(anonymous) @ angular.js:12685
(anonymous) @ angular.js:12492
(anonymous) @ angular.js:12244
(anonymous) @ angular.js:17051
(anonymous) @ angular.js:17095
$digest @ angular.js:18232
$apply @ angular.js:18531
(anonymous) @ angular.js:20448
h @ angular.js:6362
(anonymous) @ angular.js:6642
VM94:1 GET https://<GRAFANA_URL>:<GRAFANA_PORT>/api/datasources/proxy/1/query?db=telegraf&q=SELECT%20non_negative_derivative(last(%22ifHCInOctets%22)%2C%201s)%20AS%20%22In%22%2C%20non_negative_derivative(last(%22ifHCOutOctets%22)%2C%201s)%20*-1%20AS%20%22Out%20(-)%22%20FROM%20%22snmp.hosts.ifXTable%22%20WHERE%20(%22agent_host%22%20%3D~%20%2F%5E192%5C.168%5C.1%5C.254%24%2F%20AND%20%22ifName%22%20!~%20%2F%5Edocker%7Cveth%2F)%20AND%20time%20%3E%3D%201556812800000ms%20GROUP%20BY%20time(10m)%2C%20%22agent_host%22%2C%20%22ifName%22%20fill(null)&epoch=ms net::ERR_SPDY_PROTOCOL_ERROR 200
(anonymous) @ VM94:1
(anonymous) @ angular.js:12759
(anonymous) @ angular.js:12492
(anonymous) @ angular.js:12244
(anonymous) @ angular.js:17051
(anonymous) @ angular.js:17095
$digest @ angular.js:18232
(anonymous) @ angular.js:18462
h @ angular.js:6362
(anonymous) @ angular.js:6642
setTimeout (async)
i.defer @ angular.js:6640
$evalAsync @ angular.js:18460
(anonymous) @ angular.js:16923
d @ angular.js:17095
then @ angular.js:17020
t.onMetricsPanelRefresh @ metrics_panel_ctrl.ts:153
s.emit @ index.js:150
e.emit @ TagBadge.tsx:13
e.refresh @ PanelModel.ts:330
t.componentDidMount @ QueryOptions.tsx:115
Ha @ react-dom.production.min.js:216
Wa @ react-dom.production.min.js:220
(anonymous) @ react-dom.production.min.js:250
t.unstable_runWithPriority @ scheduler.production.min.js:18
Os @ react-dom.production.min.js:250
Ls @ react-dom.production.min.js:249
As @ react-dom.production.min.js:248
Ps @ react-dom.production.min.js:251
Cn @ react-dom.production.min.js:85
VM94:1 GET https://<GRAFANA_URL>:<GRAFANA_PORT>/api/datasources/proxy/1/query?db=telegraf&q=SELECT%20non_negative_derivative(last(%22ifHCInOctets%22)%2C%201s)%20AS%20%22In%22%2C%20non_negative_derivative(last(%22ifHCOutOctets%22)%2C%201s)%20*-1%20AS%20%22Out%20(-)%22%20FROM%20%22snmp.hosts.ifXTable%22%20WHERE%20(%22agent_host%22%20%3D~%20%2F%5E192%5C.168%5C.1%5C.254%24%2F%20AND%20%22ifName%22%20!~%20%2F%5Edocker%7Cveth%2F)%20AND%20time%20%3E%3D%201556812800000ms%20GROUP%20BY%20time(10m)%2C%20%22agent_host%22%2C%20%22ifName%22%20fill(null)&epoch=ms net::ERR_SPDY_PROTOCOL_ERROR 200
(anonymous) @ VM94:1
(anonymous) @ angular.js:12759
(anonymous) @ angular.js:12492
(anonymous) @ angular.js:12244
(anonymous) @ angular.js:17051
(anonymous) @ angular.js:17095
$digest @ angular.js:18232
$apply @ angular.js:18531
(anonymous) @ angular.js:18832
h @ angular.js:6362
(anonymous) @ angular.js:6642
setTimeout (async)
i.defer @ angular.js:6640
$applyAsync @ angular.js:18831
(anonymous) @ angular.js:12544
w @ angular.js:12785
y.onload @ angular.js:12702
load (async)
(anonymous) @ angular.js:12685
(anonymous) @ angular.js:12492
(anonymous) @ angular.js:12244
(anonymous) @ angular.js:17051
(anonymous) @ angular.js:17095
$digest @ angular.js:18232
$apply @ angular.js:18531
(anonymous) @ angular.js:20448
h @ angular.js:6362
(anonymous) @ angular.js:6642
setTimeout (async)
i.defer @ angular.js:6640
a @ angular.js:20438
e.startNextRefreshTimer @ emitter.ts:26
(anonymous) @ emitter.ts:1
(anonymous) @ angular.js:20440
h @ angular.js:6362
(anonymous) @ angular.js:6642

The above is under Console -> Errors. The below is under Console -> User Messages:

metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}config: undefineddata: undefinedmessage: "Network Error: undefined(undefined)"__proto__: Object
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}config: undefineddata: undefinedmessage: "Network Error: undefined(undefined)"__proto__: Object
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}config: undefineddata: undefinedmessage: "Network Error: undefined(undefined)"__proto__: Object
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}
metrics_panel_ctrl.ts:178 Panel data error: {message: "Network Error: undefined(undefined)", data: undefined, config: undefined}config: undefineddata: undefinedmessage: "Network Error: undefined(undefined)"__proto__: Objectconstructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString: ƒ toLocaleString()toString: ƒ toString()valueOf: ƒ valueOf()__defineGetter__: ƒ __defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ __lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ __proto__()set __proto__: ƒ __proto__()
4

This seems to be the underlying problem and there can be many reasons for this. Could be something to do with nginx (or similiar if you are using a proxy in front of Grafana) or a firewall or anti-virus software or something else on the server. It sounds like it could be your anti-virus software that is blocking on keywords but I’m not sure.

Do you have access to the Grafana server logs?

5

Where are the logs written to for the grafana/grafana docker?

Thanks!

6

Hi @daniellee,

My Portainer -> Grafana container logs show the following around the time of an issue occuring - Does this help? Not sure this is a proxy issue if all my other panels (on the same dashboard), seem to work ok?

2019/05/06 03:56:07 http: multiple response.WriteHeader calls

2019/05/06 03:57:07 http: multiple response.WriteHeader calls

t=2019-05-06T03:57:07+0000 lvl=eror msg="Request error" logger=context userId=1 orgId=1 uname=admin error="net/http: abort Handler" stack="/usr/local/go/src/runtime/panic.go:513 (0x435098)\n/usr/local/go/src/net/http/httputil/reverseproxy.go:284 (0xad2287)\n/go/src/github.com/grafana/grafana/pkg/api/pluginproxy/ds_proxy.go:104 (0xd2c18c)\n/go/src/github.com/grafana/grafana/pkg/api/dataproxy.go:35 (0x1006341)\n/go/src/github.com/grafana/grafana/pkg/api/api.go:263 (0x103d123)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:25 (0xa65e7c)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:17 (0xa65797)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:147 (0xa656b0)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:79 (0x79efd0)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:157 (0x770f30)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:135 (0x770c9b)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/logger.go:34 (0xa62efd)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:187 (0x7af927)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:303 (0x7aa09e)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x7a35f1)\n/usr/local/go/src/net/http/server.go:2741 (0x6b379a)\n/usr/local/go/src/net/http/server.go:1847 (0x6afb35)\n/usr/local/go/src/runtime/asm_amd64.s:1333 (0x465570)\n"

t=2019-05-06T03:57:07+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/query status=500 remote_addr=<REMOTE_IP> time_ms=60000 size=122 referer="https://<GRAFANA_URL>:<GRAFANA_PORT>/d/NyZxUIxmz/hosts-dashboard?orgId=1&fullscreen&edit&panelId=85"

t=2019-05-06T03:58:07+0000 lvl=eror msg="Request error" logger=context userId=1 orgId=1 uname=admin error="net/http: abort Handler" stack="/usr/local/go/src/runtime/panic.go:513 (0x435098)\n/usr/local/go/src/net/http/httputil/reverseproxy.go:284 (0xad2287)\n/go/src/github.com/grafana/grafana/pkg/api/pluginproxy/ds_proxy.go:104 (0xd2c18c)\n/go/src/github.com/grafana/grafana/pkg/api/dataproxy.go:35 (0x1006341)\n/go/src/github.com/grafana/grafana/pkg/api/api.go:263 (0x103d123)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:25 (0xa65e7c)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:17 (0xa65797)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:147 (0xa656b0)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:79 (0x79efd0)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:157 (0x770f30)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:135 (0x770c9b)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/logger.go:34 (0xa62efd)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:187 (0x7af927)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:303 (0x7aa09e)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x7a35f1)\n/usr/local/go/src/net/http/server.go:2741 (0x6b379a)\n/usr/local/go/src/net/http/server.go:1847 (0x6afb35)\n/usr/local/go/src/runtime/asm_amd64.s:1333 (0x465570)\n"

t=2019-05-06T03:58:07+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/query status=500 remote_addr=<REMOTE_IP> time_ms=60003 size=122 referer="https://<GRAFANA_URL>:<GRAFANA_PORT>/d/NyZxUIxmz/hosts-dashboard?orgId=1&fullscreen&edit&panelId=85"

2019/05/06 03:58:07 http: multiple response.WriteHeader calls

2019/05/06 03:59:09 http: multiple response.WriteHeader calls

t=2019-05-06T03:59:09+0000 lvl=eror msg="Request error" logger=context userId=1 orgId=1 uname=admin error="net/http: abort Handler" stack="/usr/local/go/src/runtime/panic.go:513 (0x435098)\n/usr/local/go/src/net/http/httputil/reverseproxy.go:284 (0xad2287)\n/go/src/github.com/grafana/grafana/pkg/api/pluginproxy/ds_proxy.go:104 (0xd2c18c)\n/go/src/github.com/grafana/grafana/pkg/api/dataproxy.go:35 (0x1006341)\n/go/src/github.com/grafana/grafana/pkg/api/api.go:263 (0x103d123)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:25 (0xa65e7c)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:17 (0xa65797)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:147 (0xa656b0)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:79 (0x79efd0)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:157 (0x770f30)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:135 (0x770c9b)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/logger.go:34 (0xa62efd)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:187 (0x7af927)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:303 (0x7aa09e)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x7a35f1)\n/usr/local/go/src/net/http/server.go:2741 (0x6b379a)\n/usr/local/go/src/net/http/server.go:1847 (0x6afb35)\n/usr/local/go/src/runtime/asm_amd64.s:1333 (0x465570)\n"

t=2019-05-06T03:59:09+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/query status=500 remote_addr=<REMOTE_IP> time_ms=60001 size=122 referer="https://<GRAFANA_URL>:<GRAFANA_PORT>/d/NyZxUIxmz/hosts-dashboard?orgId=1&fullscreen&edit&panelId=85"

t=2019-05-06T03:59:14+0000 lvl=eror msg="Request error" logger=context userId=1 orgId=1 uname=admin error="net/http: abort Handler" stack="/usr/local/go/src/runtime/panic.go:513 (0x435098)\n/usr/local/go/src/net/http/httputil/reverseproxy.go:284 (0xad2287)\n/go/src/github.com/grafana/grafana/pkg/api/pluginproxy/ds_proxy.go:104 (0xd2c18c)\n/go/src/github.com/grafana/grafana/pkg/api/dataproxy.go:35 (0x1006341)\n/go/src/github.com/grafana/grafana/pkg/api/api.go:263 (0x103d123)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_tracing.go:25 (0xa65e7c)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/request_metrics.go:17 (0xa65797)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/recovery.go:147 (0xa656b0)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:79 (0x79efd0)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:157 (0x770f30)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:135 (0x770c9b)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:112 (0x79f04e)\n/go/src/github.com/grafana/grafana/pkg/middleware/logger.go:34 (0xa62efd)\n/usr/local/go/src/runtime/asm_amd64.s:522 (0x46385a)\n/usr/local/go/src/reflect/value.go:447 (0x4c1223)\n/usr/local/go/src/reflect/value.go:308 (0x4c0cb3)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:177 (0x771243)\n/go/src/github.com/grafana/grafana/vendor/github.com/go-macaron/inject/inject.go:137 (0x770ba9)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/context.go:121 (0x79f130)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:187 (0x7af927)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/router.go:303 (0x7aa09e)\n/go/src/github.com/grafana/grafana/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x7a35f1)\n/usr/local/go/src/net/http/server.go:2741 (0x6b379a)\n/usr/local/go/src/net/http/server.go:1847 (0x6afb35)\n/usr/local/go/src/runtime/asm_amd64.s:1333 (0x465570)\n"

t=2019-05-06T03:59:14+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/query status=500 remote_addr=<REMOTE_IP> time_ms=60003 size=122 referer="https://<GRAFANA_URL>:<GRAFANA_PORT>/d/NyZxUIxmz/hosts-dashboard?orgId=1&fullscreen&edit&panelId=85"

2019/05/06 03:59:14 http: multiple response.WriteHeader calls
7

A panic is being caused in the middleware in Grafana due to the response from InfluxDB. Could be a bug in Grafana or could be malformed data from InfluxDB (which could be malformed due to a proxy).

This looks similar to: discret plugin and many states crashes grafana · Issue #16531 · grafana/grafana · GitHub which was caused by Nginx.

  • Are you able to try the same query in Grafana 5.x? If it works there then it probably is a bug in Grafana 6.1.4.

  • Can you see what the response from InfluxDB is?

8

I’ll try the same dashboard in Grafana 5.x when I get a chance.

Also, when you refer to the response from InfluxDB, how is best to grab this? Is it simply running the following query direct to InfluxDB CLI (outside of Grafana) and, if so, do I need to change the query somewhat to get rid of Grafana-specific $timeFilter, __interval etc?

SELECT non_negative_derivative(last(“ifHCInOctets”), 1s) AS “In”, non_negative_derivative(last(“ifHCOutOctets”), 1s) *-1 AS “Out (-)” FROM “snmp.hosts.ifXTable” WHERE (“agent_host” =~ /^host/ AND “ifName” !~ /^docker|veth/) AND timeFilter GROUP BY time(__interval), “agent_host”, “ifName” fill(null)

9

You can see the raw query in the query inspector in Grafana:

image
image2041×350 43.2 KB

as the macros like $timeFilter are Grafana concepts that are interpolated into a raw InfluxDB sql query. You can then try running the raw query directly in the InfluxDB CLI outside of Grafana.