Delay in alert state transition With Infinity data source

Grafana
,
1

Hey Guys,
I have this alert rule which will hit an endpoint to retrieve a data every 5min using Infinity data source but the endpoint return the current time data in a latency of about 2-4min.
Its a simple alert rule which should trigger alert whenever the last value is below 80 with an evaluation of 5m and a pending period of 15min.
the threshold met but the alert didn’t trigger as i checked the state history it clearly shows the transition to pending period was late I’m not sure why grafana didn’t transition it to pending state when the first instance happened as below.

image
image1339×400 96.7 KB

Ideally it should have triggered pending state when the first data point dropped but it didn’t.
As I checked the state history the alert transitioned to Normal(missing series) and after 10min but even after 10min the value was still below 80.

image
image1489×190 19.5 KB

PFB - alert rule.

image
image760×529 37.6 KB

Can anyone please help me with this what could be wrong here?

2

I think your latency is more than 2-4minutes:

obrázok
obrázok1774×1033 268 KB

Alert went to 12:28:07 to OK state, because metric was missing - but when you check history graph later, then there is some value. I would design alerting for 0-9 min metric latency. So I guess alerting didn’t see any value before 12:18 and only 12:18 has received some low value and thent instance alert went to pending state.

3

@jangaraj could you please suggest on how do we set it according to 0-9min latency?
is it by increasing the interval from 5 to 10m or any other option?

4

I don’t know for your case. Generally, use alert query, which will select last value from last 10min.