I have a set of time based measurements I store in QuestDB table and I use Grafana for visualization and alerting. I’m struggling to setup one particular alert.
The measurement I’m talking about is in a form of aggregate, cumulative values. For particular point in time T there’s a value X, in general X(T2) >= X(T1) where T2 >= T1. In other words new value is always bigger or equal than old value(s).
I want to get notified every day at most once if this day value has increased more than N. To detect that, I have a following query (named QUERY):
SELECT
now() AS time,
max(cumulative_value) - min(cumulative_value) AS value
FROM "measurement"
WHERE timestamp >= date_trunc('day', now()) AND timestamp <= now()
Then I have reducer (named REDUCER):
Function: Last
Input: QUERY
Mode: Replace Non-numeric Values
And lastly I have THRESHOLD which is an aler rule:
Input REDUCE is above N
This alert rule is evaluated every 1h, pending period is 0s. Then it’s matched via labels with notification policy that associates it with contact point. Summary is:
Daily limit (N) has been exceeded. Current value is {{ $values.REDUCE }}
Notification policy settings are visible on the screenshot:
Repeat interval is purposely set to extreme value because I want this alert to be sent at most once during the day, each day should be evaluated separetly.
The problem is, that alert is send more than once a day. I have no idea why, but alert is triggered once when result of the query exceeds defined limit (and that’s fine), but it’s also triggered one more time, several minutes after 1AM. The interesting part is that {{ $values.REDUCE }} of summary is lower than value that should trigger an alert. When alert is triggered several minutes after 1AM, the message looks like
Daily limit (15) has been exceeded. Current value is 1.2765
Which seems to be nonsense.
I suspect that it might has something to do with time zones, QuestDB timestamps are in UTC whereas Grafana picks up my timezone (which is +1), that would somehow explain why unexpected alerts are always several minutes after 1AM. But I’ve got no idea how to further troubleshoot that and how to solve this issue.
The final question is - how to set it up in the way that aler is sent at most once a day?