Hopefully Simple Grafana Alerting Issue with Prometheus data source (tsdb.HandleRequest() error invalid value type)

Grafana
1

I’m not sure what I’m doing wrong here. I have a very simple metrics scrape in prometheus to read relative humidity and temperature, which is then graphed in Grafana. No matter what I do with the alerts for any graph in my install I get the error “tsdb.HandleRequest() error invalid value type.” I’m on Grafana 4.6.0, fresh install yesterday from website stable binary builds, on Fedora 24, and Prometheus 1.8.1 on from stable channel binaries downloaded yesterday.

This is my graph config:

image
image1170×330 17.9 KB

This is the Alert config with the error:

image
image1099×617 28 KB

2 Likes
2

Can you check you grafana server side log? looks like Prometheus is returning something unexpected

3

In the grafana log, whenever I click the test button I see what’s below, but it’s just the same thing that’s in the error.

t=2017-10-29T15:35:39-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:06-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:23-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state

4

Are there debug logging flags I could enable to get more information?

5

Are you measuring the humidity for your cigars? :slight_smile: This just became much more interesting. What tech are you using?

Back to the issue…
If you arer run the query on Prometheus. is everything working as expected? ( http://localhost:9090/graph ) what kind of numbers can you see?

Would also be great if you could include the response from prometheus from your query in the panel. We have a guide for how to do that here How To Use Grafana's Query Inspector to troubleshoot issues

6

@bergquist Yes! I wrote this instructable(below) a while back about my setup. I was logging the output with elasticsearch / kibana, but it was too resource heavy for my home server, and stopped working when I upgraded Fedora. I was experimenting with Prometheus / Grafana for work so I switched it over to generate Prometheus metrics over the weekend. I’ll try and get that information when I get home tonight. If it helps, the graphs look fine, the only thing that doesn’t work is the alert part. Thanks!

7

Here’s the output from the query inspector:

{
  "xhrStatus": "complete",
  "request": {
"method": "GET",
"url": "api/datasources/proxy/1/api/v1/query_range?query=relative_humidity%7Bjob%3D%22humidor%22%7D&start=1509419841&end=1509420141&step=15"
  },
  "response": {
"status": "success",
"data": {
  "resultType": "matrix",
  "result": [
    {
      "metric": {
        "__name__": "relative_humidity",
        "instance": "humidor.sensor:443",
        "job": "humidor"
      },
      "values": [
        [
          1509419841,
          "67"
        ],
        [
          1509419856,
          "67.1"
        ],
        [
          1509419871,
          "67"
        ],
        [
          1509419886,
          "67"
        ],
        [
          1509419901,
          "67"
        ],
        [
          1509419916,
          "67"
        ],
        [
          1509419931,
          "67"
        ],
        [
          1509419946,
          "67"
        ],
        [
          1509419961,
          "67"
        ],
        [
          1509419976,
          "67"
        ],
        [
          1509419991,
          "67"
        ],
        [
          1509420006,
          "67"
        ],
        [
          1509420021,
          "67"
        ],
        [
          1509420036,
          "67"
        ],
        [
          1509420051,
          "67"
        ],
        [
          1509420066,
          "67"
        ],
        [
          1509420081,
          "67"
        ],
        [
          1509420096,
          "67"
        ],
        [
          1509420111,
          "67"
        ],
        [
          1509420126,
          "67"
        ],
        [
          1509420141,
          "67"
        ]
      ]
    }
  ]
}
  }
}
8

Also, here’s what the metrics look like coming from the device:

## Humidor Readings 
relative_humidity 67 1509420263856
temperature{units="celsius"} 20.7 1509420263856
temperature{units="fahrenheit"} 69.25999999999999 1509420263856
lid_closed 1
9

Anyone? @bergquist? @torkel?

10

Hi i encountered the same error message with Prometheus and Grafana, is there any data that I can provide that would help resolve this issue?

Thank you

1 Like
11

We are suffering from this issue too. We cannot set up any alert on prometheus based metrics.

1 Like
12

Are you on either of the same versions above?

13

Yes grafana 4.6.0 and prometheus 1.8.1, running them in Docker.

1 Like
14

So, anyone have any idea here? @bergquist? @torkel?

15

This problem is to be fixed soon

https://github.com/grafana/grafana/issues/9777

Comes from Grafan 4.6 with Prometheus.

In the discussion patch to be expected soon