Failed to evaluate queries and expressions: input data must be a wide series but got type long (input refid)

Hello! I’m trying to setup notifications for loki source and the same query that works in “Explore” window doesn’t work in “Alert rules” window, I couldn’t find any info on this on the internet

So the query is {container="jetton-backend"} | json | status_code > 400

Here it is working in explore

And here is the same query that doesn’t work in alerts

Can anyone help please? Is there something i’m missing?

2 Likes

I have the same problem with alerting on logs with Loki & Promtail
Working on explore but not in alert.
Query: {namespace=“monitoring”} |= “0”
Response: “Failed to evaluate queries and expressions: input data must be a wide series but got type long (input refid)”

Any help would be apreciated

I was trying to figure this out too. From this Github issue I get the impression that Grafana alerting can only handle numeric data, not log messages.

So you can do a count of log messages matching a query, alert when the count is > 0, and then manually go looking through the logs when you get an alert, but as far as I can see there’s no way to actually get the log message contents.

Which seems nuts, so hopefully I’m missing something and someone else will chime in here with a better solution.

count(rate({container="jetton-backend"}} | json | status_code > 400 [$__interval]))
2 Likes

@sam2b3e is correct. Grafana Alerting works on numeric data, and Loki alerts are expected to use count or count_over_time.

1 Like

Many thanks for confirming, it’s good to know I’m not missing the obvious :slight_smile:

I come from a world that uses Graylog and Elasticsearch for logging, so I’m used to being able to get log message contents in alerts, and it’s one of the first things I set out to do when evaluating Loki. I’m genuinely rather surprised that it can’t be done.

Is there anything in the roadmap that might address this in the future? Some method of having the alert notification template reference the underlying log data? I imagine it’d be a huge enabler for other people in my situation looking to move their logging over Loki (which in every other way is extremely compelling).

Heyy!! Guys I am also having same kind of problem,i am not able to fix that can anyone please rephrase here how to solve or give me the proper conclusion regarding with this please.

The trouble is when I’m using keywords to filter error logs use of few query that works in explore feature but not works in alert rules how to resolve this kind of stuffs

Query: {job=“sample”} |= “error” |=“https failed”

ERROR: Failed to evaluate queries and expressions: input data must be a wide series but got type long

Hi @here

Can you please us the following:

1- Is this reproducible only in a Containerized environment e.g. Docker, Kubernetes OR is reproducible on any local setup with Grafana + Loki?

2- Is it reproducible with the current stable version of Grafana 10.x?

To answer point number two, I have reproduced this using grafana/grafana-oss:10.1.4-ubuntu (Docker)

Hi @flyingpork15

What is your Data source?

Looks like Microsoft SQL to me.

1 Like

Correct

You could do a count for how many of those log entries you get and base a threshold off of that

1 Like

Hi @usman.ahmad,

Have you been able to find out if we can alert on the error message (word) vs using “count”. Just wondering if we can capture the line that contains the word error to help assist with troubleshooting/correlation.
{job=“syslog”, host=~“.±MAAS-001”, message=~“(?i).critical.”}[5m] OR
{job=“syslog”, host=~“.±MAAS-001”} |~ “critical”

Hi @maaslumen1,

Welcome to the :grafana: community support forums !!

We are excited that you joined our OSS community. Please read about some of the FAQs in the community :slight_smile:

Unfortunaitely, I did not know about this and also I am not tracking much on alerting these days.

I would recommend that you create a new post in the Alerting Category as ask your question there by providing all the details.