I have several Loki-managed alerts, as I’ve understood from the documentation these alerts expressions must be in the format of either rate() or count_over_time(), obviously when using this format the {{ $value }} variable used in the alert’s summary/description returns the value of count_over_time() and rate().
I have two questions:
1- Is it possible to use expressions without rate or count_over_time? When I use them the state of that alert is either error or nodata but I want to know is there any workaround or solution for this?
2- The reason I need the previous question is that being notified that there are some error or warning logs is not enough for me, I want to be able to pass some variables such as uid & iid (VM instance ID). Since these labels change dynamically I can’t write a static summary for them hence, is there any way to use these labels as variables such as {{ $labels.uid }}?
I’m having a similar issue, did you ever get this working?
We’re using the AWS Lambda promtail to forward lambda logs from AWS to a Grafana Labs/Cloud Loki instance, I’d like to use a label of: {{ $labels.__aws_cloudwatch_log_group }}label in my alerts to give the receiver some context around which lambda is firing the alert. My query is broad to find things like a Lambda timeout, I don’t want to have to create a query/alert per function just to inject this information… surely it’s possible?
Hello Keegs, unfortunately, I was unable to make it work, we abandoned the task and are currently in the process of developing some sort of middleware for it.
That’s all the time the same questions for everybody and the documentation for this point is clearly unacceptable. Grafana documentation in only here to say “yeah, we have documentation”, but is not ok, when documentation is not understandable, it’s not documentation.
Just circling back to this as I ended up raising a support query and looks like no one from Grafana is going to come here with an actual resolution, so for future Grafana users that hit this problem:
In my case I was using a query like this for my alert rule sum(rate(({label_1="my_label"} |="some_text_to_filter" | json) [5m]))
I had an issue as OP found in ‘2.’ where you can’t access any labels on these logql queries, everything just resolves to [no value] which makes your alerts pretty meaningless.
What Support came back with was to add without (label_2) to the end of the query, for whatever reason this then allows me to view all other labels in my alerts. I can now add context into the summary/description etc by using label variables {{ $labels.label_3}}
For reference my example logql query above now becomes: sum(rate(({label_1="my_label"} |="some_text_to_filter" | json) [5m])) without (label_2)
I’ve gone back to support to ask why we need to remove a label for any others to correctly show up, will update if I ever hear back.
The reason you are not seeing label with your query is because of the sum function. If you remove the sum function you’ll see that the rate function returns a set of labels.
What’s suggested does work (adding without), but you can also just explicitly tell the sum function to aggregate by certain labels, thereby preserving them. This may give you additional benefit of only getting the set of labels you care about, rather than everything except the one specified by without.
For example:
sum by (label2, label3) (rate(
({label1="value1"} |= "search_string")
[$__interval])
)
