Hello,
I am trying to create an alert for no data only using a mysql data source. We have a server that runs specific jobs during business hours only and writes the results to a database. If the jobs fail then no data is written to the db, which is currently what we are looking to get alerted on.
How the Alert is currently set up (Input A):
SELECT
status AS job_results,
IF(status != NULL, 0, 1) AS fakeStat
FROM table
WHERE jobName like "%JOBNAME_HERE%" AND dateadded > date(now())
Alert Condition: Threshold Input A is above 1
I created a fake metric (fakeStat) and set the threshold to trigger an alert if the value is ever over 1. From there, I added the option to Alert us if no data is received. We should never receive an alert for the fake metric. The database is written to every 10 minutes, so we have an evaluation interval of 10 minutes for 1h. We also added a filter to only pull jobs from today date.
These jobs only run between m-f from 9a to 5p and I have setup mute timings accordingly. There is no data being written to the db outside of these hours. The problem That I am running into is every morning, we are getting an alert that no data has been received because even though we have mute timing setup, its looking at the non business hours as well.
Is there anyway to ignore when no data is received outside of business hours?
The footer of the alert reads: Observed 4h0m35s before this notification was delivered, at 2023-06-28 X:X:X X:X:X MST
Running Grafana v9.5.2