Hopefully Simple Grafana Alerting Issue with Prometheus data source (tsdb.HandleRequest() error invalid value type)

I’m not sure what I’m doing wrong here. I have a very simple metrics scrape in prometheus to read relative humidity and temperature, which is then graphed in Grafana. No matter what I do with the alerts for any graph in my install I get the error “tsdb.HandleRequest() error invalid value type.” I’m on Grafana 4.6.0, fresh install yesterday from website stable binary builds, on Fedora 24, and Prometheus 1.8.1 on from stable channel binaries downloaded yesterday.

This is my graph config:

This is the Alert config with the error:

2 Likes

Can you check you grafana server side log? looks like Prometheus is returning something unexpected

In the grafana log, whenever I click the test button I see what’s below, but it’s just the same thing that’s in the error.

t=2017-10-29T15:35:39-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:06-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:23-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state
t=2017-10-29T15:36:24-0400 lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalHandler ruleId=0 name=“Humidity alert” error=“tsdb.HandleRequest() error invalid value type” changing state to=keep_state

Are there debug logging flags I could enable to get more information?

Are you measuring the humidity for your cigars? :slight_smile: This just became much more interesting. What tech are you using?

Back to the issue…
If you arer run the query on Prometheus. is everything working as expected? ( http://localhost:9090/graph ) what kind of numbers can you see?

Would also be great if you could include the response from prometheus from your query in the panel. We have a guide for how to do that here How To Use Grafana's Query Inspector to troubleshoot issues

@bergquist Yes! I wrote this instructable(below) a while back about my setup. I was logging the output with elasticsearch / kibana, but it was too resource heavy for my home server, and stopped working when I upgraded Fedora. I was experimenting with Prometheus / Grafana for work so I switched it over to generate Prometheus metrics over the weekend. I’ll try and get that information when I get home tonight. If it helps, the graphs look fine, the only thing that doesn’t work is the alert part. Thanks!

Here’s the output from the query inspector:

{
  "xhrStatus": "complete",
  "request": {
"method": "GET",
"url": "api/datasources/proxy/1/api/v1/query_range?query=relative_humidity%7Bjob%3D%22humidor%22%7D&start=1509419841&end=1509420141&step=15"
  },
  "response": {
"status": "success",
"data": {
  "resultType": "matrix",
  "result": [
    {
      "metric": {
        "__name__": "relative_humidity",
        "instance": "humidor.sensor:443",
        "job": "humidor"
      },
      "values": [
        [
          1509419841,
          "67"
        ],
        [
          1509419856,
          "67.1"
        ],
        [
          1509419871,
          "67"
        ],
        [
          1509419886,
          "67"
        ],
        [
          1509419901,
          "67"
        ],
        [
          1509419916,
          "67"
        ],
        [
          1509419931,
          "67"
        ],
        [
          1509419946,
          "67"
        ],
        [
          1509419961,
          "67"
        ],
        [
          1509419976,
          "67"
        ],
        [
          1509419991,
          "67"
        ],
        [
          1509420006,
          "67"
        ],
        [
          1509420021,
          "67"
        ],
        [
          1509420036,
          "67"
        ],
        [
          1509420051,
          "67"
        ],
        [
          1509420066,
          "67"
        ],
        [
          1509420081,
          "67"
        ],
        [
          1509420096,
          "67"
        ],
        [
          1509420111,
          "67"
        ],
        [
          1509420126,
          "67"
        ],
        [
          1509420141,
          "67"
        ]
      ]
    }
  ]
}
  }
}

Also, here’s what the metrics look like coming from the device:

## Humidor Readings 
relative_humidity 67 1509420263856
temperature{units="celsius"} 20.7 1509420263856
temperature{units="fahrenheit"} 69.25999999999999 1509420263856
lid_closed 1

Anyone? @bergquist? @torkel?

Hi i encountered the same error message with Prometheus and Grafana, is there any data that I can provide that would help resolve this issue?

Thank you

1 Like

We are suffering from this issue too. We cannot set up any alert on prometheus based metrics.

1 Like

Are you on either of the same versions above?

Yes grafana 4.6.0 and prometheus 1.8.1, running them in Docker.

1 Like

So, anyone have any idea here? @bergquist? @torkel?

This problem is to be fixed soon

https://github.com/grafana/grafana/issues/9777

Comes from Grafan 4.6 with Prometheus.

In the discussion patch to be expected soon