Grafana Alert Fires continuously even when the condition at threshold is not satisfied

Grafana Alerting
1

  • What Grafana version and what operating system are you using? “grafana:12.0.0; Linux”

  • What are you trying to achieve? “Alert only when threshold condition is satisfied“

  • How are you trying to achieve it? “Configure Alert rule“

  • What happened? “Alert firing, even when the threshold condition is not met“

  • What did you expect to happen? “Alert only when threshold condition is met“

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were. “No Errors found in the log”

    Alert rule setting

    Query A
    count by (job, instance) (target_info{group=“KCA”, process_pid!=“”} offset 24h)

    Query B
    count by (job, instance) (target_info{group=“KCA”, process_pid!=“”})
    OR
    (count by (job, instance) (target_info{group=“KCA”, process_pid!=“”} offset 24h) * 0)

    Math C
    $A-$B

    Threshold D IS Above 0

    with following expression
    MY Analytics App SOME instances Down - A=1.000000, B=1.000000, C=0.000000, D=0.000000

Actual Alert Rule Config is

apiVersion: 1
groups:
- orgId: 1
name: kca-instance-count
folder: TTK Analytics Apps
interval: 5m
rules:
- uid: ef63d6b4yppmod
title: TTK Analytics App KCA instances Down
condition: B
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: mimir
model:
datasource:
type: prometheus
uid: mimir
editorMode: code
expr: count by (job, instance) (target_info{group=“KCA”} offset 24h)
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: A
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: mimir
model:
datasource:
type: prometheus
uid: mimir
editorMode: code
expr: “count by (job, instance) (target_info{group="KCA"})\r\nOR\r\n(count by (job, instance) (target_info{group="KCA"} offset 24h) * 0)”
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: B
- refId: C
datasourceUid: expr
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: 

reducer:
params: 

type: avg
type: query
datasource:
name: Expression
type: expr
uid: expr
expression: round($A - $B)
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: math
- refId: D
datasourceUid: expr
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: 

reducer:
params: 

type: avg
type: query
datasource:
name: Expression
type: expr
uid: expr
expression: C
intervalMs: 1000
maxDataPoints: 43200
refId: D
type: threshold
noDataState: NoData
execErrState: Error
for: 5m
keepFiringFor: 5m
labels:
alert_type: instance_count_decreased
group: KCA
severity: critical
team: KCA
isPaused: false
  • Did you follow any online instructions? If so, what is the URL? “NO“