Hello! I’m trying to setup notifications for loki source and the same query that works in “Explore” window doesn’t work in “Alert rules” window, I couldn’t find any info on this on the internet
So the query is {container="jetton-backend"} | json | status_code > 400
I have the same problem with alerting on logs with Loki & Promtail
Working on explore but not in alert.
Query: {namespace=“monitoring”} |= “0”
Response: “Failed to evaluate queries and expressions: input data must be a wide series but got type long (input refid)”
I was trying to figure this out too. From this Github issue I get the impression that Grafana alerting can only handle numeric data, not log messages.
So you can do a count of log messages matching a query, alert when the count is > 0, and then manually go looking through the logs when you get an alert, but as far as I can see there’s no way to actually get the log message contents.
Which seems nuts, so hopefully I’m missing something and someone else will chime in here with a better solution.
Many thanks for confirming, it’s good to know I’m not missing the obvious
I come from a world that uses Graylog and Elasticsearch for logging, so I’m used to being able to get log message contents in alerts, and it’s one of the first things I set out to do when evaluating Loki. I’m genuinely rather surprised that it can’t be done.
Is there anything in the roadmap that might address this in the future? Some method of having the alert notification template reference the underlying log data? I imagine it’d be a huge enabler for other people in my situation looking to move their logging over Loki (which in every other way is extremely compelling).
Heyy!! Guys I am also having same kind of problem,i am not able to fix that can anyone please rephrase here how to solve or give me the proper conclusion regarding with this please.
The trouble is when I’m using keywords to filter error logs use of few query that works in explore feature but not works in alert rules how to resolve this kind of stuffs
Query: {job=“sample”} |= “error” |=“https failed”
ERROR: Failed to evaluate queries and expressions: input data must be a wide series but got type long
Have you been able to find out if we can alert on the error message (word) vs using “count”. Just wondering if we can capture the line that contains the word error to help assist with troubleshooting/correlation.
{job=“syslog”, host=~“.±MAAS-001”, message=~“(?i).critical.”}[5m] OR
{job=“syslog”, host=~“.±MAAS-001”} |~ “critical”