Different thresholds for an alert for different time periods

Grafana Alerting
1

For an alert in Grafana, can you set different thresholds for different time periods ?
We are using Prometheus as the data source.
Thanks

2

Hi,

By time periods do you mean time of day (hours) or months or something like that?
You could move the heavy lifting to prometheus query to return data only when the metric reaches thresholds in given time and return no data when it doesn’t (or do 0 / 1 with bool operator). Sample query that limits the metric to hours of the day:

image
image1129×470 35 KB

(notice time zones while working with hours).

You can replace vector(n) with a metric query that would return 0 / 1 when the threshold is breached:

image
image1129×470 29.7 KB

To put it all together it could look like:

image
image1129×470 38.1 KB
r

It would return 1 when there’s more than 4300 ups after 12 o’clock (timezones!) or if there’s more than 4270 ups before 12 o’clock. Otherwise, it would return 0. Now a simple threshold if the query is above 0 and you should be good to go.

A few considerations:

  • are the timings constant?
  • are you sure about the thresholds in those hours? Maybe someone will be working late?
  • the query might be slightly more complicated, so it might become unreadable. You might also have some troubles using values and labels in templates (if you’re planning to).
  • you could also achieve something similar (theoretically) by creating two alerts and muting them in the times they shouldn’t be firing but there’re more considerations for them (what if the alert fires right when the muting is coming in effect, what with the timings, if the alert fires a minute before it should, will the notification come right away or will you have to wait for Grafana alerts manager to send it again?)

Hope that helps!