Grafana 7.0.1 alerting heart not updating?

Hi!

Just recently upgraded to 7.0.1 on our test instance and I’m noticing that all of our alert panels have a red heart even if when testing the rule it displays an “ok” condition. Is this a bug or is there a new setting that needs to be changed in the config to correct this?



image

Anyone have any ideas? I created a new panel and the alerting doesn’t change from the grey heart on that panel. Real odd. is this a known issue?

The red heart, it means your data still in the alert condition and you need to rectified until the data is out of alert threshold.
I didn’t the the grey heart in the pictures that you sent. Please send the condition that hit the grey condition.

Regards,
Fadjar Tandabawana

But you can see it isn’t in the alert condition. The alert condition is if the min of query a is below 150 between 15m ago and now. in the image you can see that query a, which is A temp from the first image, has a min over the last several hours of 1.415k.

The red heart is showing even though it when I test the rule (bottom picture) it evaluates as “OK”.

Try to update to 7.0.3, there’s a lots of improvement.

I tried. Unfortunately it is still occurring.

I will try to clear the alert states present in the mySQL db and see if that fixes things.

I saw something weird in the image of the Alert that you have…

Evaluate every: 5m for 0m
Is it right?

I believe so? It should evaluate every 5m but it doesn’t need to match a specific time length for how long it needs to stay in alert to be satisfied. Basically, I want the alert to check the min value for the query and see if it is below 150 every 5 mins. If it is below, go into alert, if not no alert. I don’t want it to wait any amount of time to reevaluate again to make sure it is still in an alert condition.

Try to change like below:

Evaluate every 20s for 1m
Conditions:
When min() of query (a,5m,now) is below 2000
This is just for the test…

Then check the rule

Let’s see the result…



image

The heart still isn’t updating.

That’s firing… Wait until the time evaluation period in the for 1m finished…

It isn’t changing from pending. It should only take 1 minute I believe? I don’t think it is working properly.

See the query (c,15m,now)… The condition still need to wait until 15m finished, so change to 5m

OH I see ok. I can’t, unfortunately our data is on a time delay and usually is about 10-12m behind. Ok I will recheck in 15 minutes to see what it does.

If your data have behind 10-12m, just change the query option like this:

And you still can reduce the time to 5 m in the conditions

My metric query panel looks different since we use OpenTSDB.

Check the Query Options, right side of the datasource…
Open it and set the timeshift

Oh I found it ok, I adjusted that and set to 5m timeperiod in the alerting.

I changed it from 20s for 1m to just 20s, the for 1m was causing it to hang on pending.
Still, it is alerting but the heart does not change color to red.

image

Back to the dashboard, not in the edit page, and refresh the page using F5…