After upgrading from 8.1.3 to 8.1.5 eror msg="Alert Panic"

After upgrading from 8.1.3 to 8.1.5, the liter is filled with messages approximately every 30 seconds.

grafana grafana-server[879]: t=2021-09-27T09:22:21+0300 lvl=eror msg="Alert Panic" logger=alerting.engine error="runtime error: index out of range [1] with length 1" stack="[engine.go:177 panic.go:965 panic.go:88 response_parser.go:117 regexp.go:699 regexp.go:634 regexp.go:698 response_parser.go:102 response_parser.go:81 response_parser.go:41 influxdb.go:158 core_plugin.go:81 manager.go:301 instrumentation.go:40 instrumentation.go:69 manager.go:300 data_plugin_adapter.go:63 tsdb.go:242 service.go:80 query.go:167 query.go:52 eval_handler.go:37 engine.go:188]"

I suppose that the problem is somehow related to warnings, but I cannot understand one or all of them. I would be grateful for your help.

Hello,

I had a similar errors for that version (v8.1.5 (c725a4259a)) and it turns out that for me 1 alert did not work (you can find it using the Test Alert option). The reason the Alert was not working was that the query “Alias” field name. Apparently that now throws as exception in alerting. My name was core 1: $col
Anything else does work, even the other 3 cores with similar aliases.
So I changed the aliad to core1: $col and that works.

However since this is only a workaround and not a resolution I exported my dasnboard, stopped grafana.
Then I dropped the sqlite3 DB for grafana and started grafana.
This recreates the DB without the dashboard.
I then imported the dashboard and the exception was back.
I will not perform a clean install as my workaround does work.
Perhaps it will help you as well.

Logging exceptions for me:

grafana            | t=2021-09-29T18:33:27+0200 lvl=eror msg="Alert Panic" logger=alerting.engine error="runtime error: index out of range [1] with length 1" stack="[engine.go:177 panic.go:965 panic.go:88 response_parser.go:117 regexp.go:699 regexp.go:634 regexp.go:698 response_parser.go:102 response_parser.go:81 response_parser.go:41 influxdb.go:158 core_plugin.go:81 manager.go:301 instrumentation.go:40 instrumentation.go:69 manager.go:300 data_plugin_adapter.go:63 tsdb.go:242 service.go:80 query.go:167 query.go:52 eval_handler.go:37 engine.go:188]"
grafana            | t=2021-09-29T18:31:51+0200 lvl=eror msg="Request error" logger=context userId=1 orgId=1 uname=mark error="runtime error: index out of range [1] with length 1" stack="/usr/local/go/src/runtime/panic.go:88 (0x43f124)\n/drone/src/pkg/tsdb/influxdb/response_parser.go:117 (0x23bd31a)\n/usr/local/go/src/regexp/regexp.go:699 (0x77481d)\n/usr/local/go/src/regexp/regexp.go:634 (0x76e842)\n/usr/local/go/src/regexp/regexp.go:698 (0x76f15c)\n/drone/src/pkg/tsdb/influxdb/response_parser.go:102 (0x23bbff4)\n/drone/src/pkg/tsdb/influxdb/response_parser.go:81 (0x23bb7d7)\n/drone/src/pkg/tsdb/influxdb/response_parser.go:41 (0x23baeeb)\n/drone/src/pkg/tsdb/influxdb/influxdb.go:158 (0x23b3d7d)\n/drone/src/pkg/plugins/backendplugin/coreplugin/core_plugin.go:81 (0x181b4fa)\n/drone/src/pkg/plugins/backendplugin/manager/manager.go:301 (0x1a9e456)\n/drone/src/pkg/plugins/backendplugin/instrumentation/instrumentation.go:40 (0x1a98b8f)\n/drone/src/pkg/plugins/backendplugin/instrumentation/instrumentation.go:69 (0x1a9ba77)\n/drone/src/pkg/plugins/backendplugin/manager/manager.go:300 (0x1a9b97a)\n/drone/src/pkg/tsdb/data_plugin_adapter.go:63 (0x19441e7)\n/drone/src/pkg/plugins/tsdb.go:242 (0x1943c53)\n/drone/src/pkg/tsdb/service.go:80 (0x1943b23)\n/drone/src/pkg/services/alerting/conditions/query.go:167 (0x2221e32)\n/drone/src/pkg/services/alerting/conditions/query.go:52 (0x2220c87)\n/drone/src/pkg/services/alerting/eval_handler.go:37 (0x1424a97)\n/drone/src/pkg/services/alerting/test_rule.go:36 (0x142fff0)\n/drone/src/pkg/api/alerting.go:137 (0x214f5bc)\n/usr/local/go/src/reflect/value.go:476 (0x4afa46)\n/usr/local/go/src/reflect/value.go:337 (0x4aef18)\n/drone/src/pkg/macaron/inject.go:164 (0x10743f6)\n/drone/src/pkg/macaron/inject.go:124 (0x1073d71)\n/drone/src/pkg/api/routing/routing.go:18 (0x19654cd)\n/usr/local/go/src/reflect/value.go:476 (0x4afa46)\n/usr/local/go/src/reflect/value.go:337 (0x4aef18)\n/drone/src/pkg/macaron/inject.go:164 (0x10743f6)\n/drone/src/pkg/macaron/inject.go:124 (0x1073d71)\n/drone/src/pkg/macaron/context.go:77 (0x1072c52)\n/drone/src/pkg/macaron/context.go:67 (0x1a60085)\n/drone/src/pkg/middleware/request_metrics.go:55 (0x1a60070)\n/usr/local/go/src/reflect/value.go:476 (0x4afa46)\n/usr/local/go/src/reflect/value.go:337 (0x4aef18)\n/drone/src/pkg/macaron/inject.go:164 (0x10743f6)\n/drone/src/pkg/macaron/inject.go:124 (0x1073d71)\n/drone/src/pkg/macaron/context.go:77 (0x1072c52)\n/drone/src/pkg/macaron/context.go:67 (0x1a5ff04)\n/drone/src/pkg/middleware/recovery.go:166 (0x1a5fef7)\n/drone/src/pkg/macaron/context.go:35 (0x1072a32)\n/drone/src/pkg/macaron/inject.go:144 (0x10740f1)\n/drone/src/pkg/macaron/inject.go:122 (0x1073e64)\n/drone/src/pkg/macaron/context.go:77 (0x1072c52)\n/drone/src/pkg/macaron/context.go:67 (0x1a5dfc9)\n/drone/src/pkg/middleware/logger.go:35 (0x1a5df9d)\n/usr/local/go/src/reflect/value.go:476 (0x4afa46)\n/usr/local/go/src/reflect/value.go:337 (0x4aef18)\n/drone/src/pkg/macaron/inject.go:164 (0x10743f6)\n/drone/src/pkg/macaron/inject.go:124 (0x1073d71)\n/drone/src/pkg/macaron/context.go:77 (0x1072c52)\n/drone/src/pkg/macaron/context.go:67 (0x1a611d7)\n/drone/src/pkg/middleware/request_tracing.go:56 (0x1a611ca)\n/usr/local/go/src/reflect/value.go:476 (0x4afa46)\n/usr/local/go/src/reflect/value.go:337 (0x4aef18)\n/drone/src/pkg/macaron/inject.go:164 (0x10743f6)\n/drone/src/pkg/macaron/inject.go:124 (0x1073d71)\n/drone/src/pkg/macaron/context.go:77 (0x1072c52)\n/drone/src/pkg/macaron/router.go:187 (0x10817d0)\n/drone/src/pkg/macaron/router.go:294 (0x107c34f)\n/drone/src/pkg/macaron/macaron.go:192 (0x1075bac)\n/usr/local/go/src/net/http/server.go:2887 (0x724a42)\n/usr/local/go/src/net/http/server.go:1952 (0x71ff6c)\n/usr/local/go/src/runtime/asm_amd64.s:1371 (0x47c2c0)\n"

1 Like

Thank you, buddy, for guiding you in the right direction!
I have analyzed all my rules and confirm that where the alias in its meaning contains 1 with a space, these problems arise.