Grafana 8 alerting - unable to create alert for no Data

  • What Grafana version and what operating system are you using?
    Using grafana version 8.1.2 installed as a docker container on ec2 instance.

  • What are you trying to achieve?
    Create alert when no data is coming

  • How are you trying to achieve it?
    Using grafana 8 alerting system

  • What happened?
    Getting error - “Input data must be wide series got type not (input refid)”

  • What did you expect to happen?
    For no data case - alert should show.

  • Can you copy/paste the configuration(s) that you are having problems with?
    This is the query i want to run:

alert condition:

also tried with Math/Reduce & Math inplace of classic condition - still gets same error. But if data is coming then no alert works fine.

Hoping to get early response.

4 Likes

I would like to add that the same happens for MySQL when counting aggregated rows, expecing to alert on >0 in response.

I feel that this should trigger the “No Data” situation for an alert. The error is likely coming from the expression, however the expression does not accept the option for “Result of B has no value.”

Am I correct in reading this to mean that you’re getting an error with the query when you try to save the alert rule or preview alerts?

Can you clarify what you mean? The NoData state has a very specific definition and I also want to make sure you’re aware of that State and Health of alerting rules | Grafana Labs

@davidparrott
Hello.
I have the same problem. Use datasoure influxdb.
If have no data, Alerting is not working.
Need Alert if no Data.


Screenshot_31
Screenshot_32

@chrsergey

Have you found a solution for your problem yet? I recently changed to the new “unified alerting” and I am facing the same problem.

@sebbleroni
No, I havent solutions.

@davidparrott @chrsergey

I think the problem could be connected to the message content itself. At least for me (I am using Telegram), grafana seems to attempt to send a message, but ultimately fails to do so (see my logs):

t=2021-12-06T12:36:31+0000 lvl=info msg="sending telegram notification" logger=alerting.notifier.telegram chat_id=991107053
t=2021-12-06T12:36:31+0000 lvl=eror msg="Failed to send webhook" logger=alerting.notifier.telegram error="Webhook response status 400 Bad Request" webhook="Test Telegram Alert"
t=2021-12-06T12:36:31+0000 lvl=info msg="level=error component=dispatcher msg=\"Notify for alerts failed\" num_alerts=1 err=\"Test Telegram Alert/telegram[0]: notify retry canceled due to unrecoverable error after 1 attempts: Webhook response status 400 Bad Request\"" logger=alertmanager org=1

The problem can be “fixed” by setting a custom telegram message:

However, now only this message without any information about the alert itself is displayed.

Also encountering this error.

The query works fine when there is data, but as soon as there is no data returned by the query then we hit the same error message: “Input data must be a wide series but got long (input refid)”

The whole point of the alert is to trigger when there is no data returned, so how can we put it in wide series format??

We’re using a SQL datasource.

3 Likes

I should also point out that the exact same query works fine in Grafana 7…

Idem, same problem here, Could not find a workaround yet.

1 Like

I’m facing this issue too, using BigQuery datasource. Grafana version 8.4.1

1 Like

Exactly the same issue here. Using AWS CloudWatch Logs as data source, to query Error Logs from our Database. But when there are no results (no error logs) the alert breaks.

Specifying the “Alert state if execution error or timeout” seems to omit the problem for us. Meaning, the alert itself is still in “Normal” state, however the “health” of the alert is “ERROR” due to this issue “input data must be a wide series but got type not (input refid)”

Running Grafana Cloud, v8.4.5-56740

UPDATE: nope, speciying the “Alert state if execution error or timeout” does not fix this issue for us.

1 Like

Still no update on this?

1 Like

Testing locally a conversion from legacy Grafana alerts to unified alerting is also causing this issue for me. Using AWS Cloudwatch to query

1 Like

Same problem here. It seems to have something to do with the format of the input data. To have more control over the input data i’m using the CSV datasource now. I’ve tried all kinds of CSV content, but no luck so far…

Similar problem.

  1. We use PostgreSQL as a source
  2. send to SLASK.
  3. When trying to set up an Alert for NO DATA notification, we get a message.
1 Like

Finally found a way to do it, using strcontains and sum functions.
This generates a continuous data set, with 0s when pattern is not found.

fields strcontains(@message, 'Failed to process upserts for table') as is_my_pattern
| stats sum(is_my_pattern) as my_pattern_count by bin(1m) as ts
| sort my_pattern_count desc
1 Like

This topic was automatically closed after 365 days. New replies are no longer allowed.