I’m using Loki to send a logs from the same app, but running in a different VM for every customer ({app="my-app", customer="customer-1"}), can I create a single heartbeat grafana alert that trigger if a customer log stops been sent? I didn’t want to create an alert per customer.
Hi,
In theory, you could count the number of logs from given consumer in time (e.g. range query 10 minutes if that’s a thing in Loki) and add > 0 condition (again if that’s a thing with loki), then use Reduce to count number of points - since you know how many points there should be (e.g. 10 minutes range query with 1 minute resolution / step), if there’s less than 10 points, you can alert (I’d go for e.g. 3 minute pending period with 1 minute evaluation).
It could look something like this
in PromQL (I’m not sure unfortunately if / how to do that in Loki).
To route the notifications you can create notification policies based on customer label or have your own service handling those in some custom way.
1 Like
Thanks for your answer. For my use case it’s not enough because I don’t want to worry about manually incrementing the number of customers in the alert.
It would be perfect if I can get the count of distinct customer’s labels in the threshold reduction, but this feature isn’t available.
I ended up solving this problem by creating a CLI with typer that deploy a customer app and uses the grafana provisioned API to create the heartbeat alerts.