There are two alerts I want to create. The first one is to check if Alloy is not working for instances with names starting from VPC-TEST-001 to VPC-TEST-020 and VPC-NEW-001 to VPC-NEW-030. I used the absent function like this: absent(up{instance="VPC-TEST-001"}) or absent(up{instance="VPC-TEST-002"}) or ... and defined all the instances. However, I want to create a more dynamic way to avoid defining each one manually.
I can use something like this:
absent(up{instance=~"VPC-TEST-00[1-9]|VPC-TEST-0[1-9][0-9]|VPC-NEW-00[1-9]|VPC-NEW-0[1-9][0-9]"})
But in this case, I’ll get an alert indicating that some instances are not in the up state, but it won’t show which one specifically.
The second alert is related to NO DATA alerts. I have a rule that collects information from synthetic checks, and I have a recording rule called test:instance_job_severity:probe_success:mean5m. Here is the rule:
(sum without(probe, config_version) (rate(probe_all_success_sum[5m]) * on(instance, job, probe) group_left(alert_sensitivity) max by(instance, job, probe, alert_sensitivity) (sm_check_info{alert_sensitivity!=""})) / sum without(probe, config_version) (rate(probe_all_success_count[5m]) * on(instance, job, probe) group_left(alert_sensitivity) max by(instance, job, probe, alert_sensitivity) (sm_check_info{alert_sensitivity!=""}))) * 100
Using this rule, I’m creating alerts, but the problem arises when I want to receive alerts for the NO DATA state. Here’s how it works: for example, I have 4 checks, and each one has an label with alert_sensitivity = HIGH. When one of them enters the NO DATA state, since there are still 3 checks with that label, the alert doesn’t go into the NO DATA state. I’ve created specific alerts for NO DATA using the absent() function, but for that, I have to manually write each check. When adding a new check, I also need to manually add that one as well. Can someone suggest how I can create an alert to receive NO DATA alerts that identify which check is in the NO DATA state, or how to create a new specific alert for that?