Grafana Alert triggering with unexpected value, under false conditions

Grafana Alerting
1

Hi,

I’m trying to explain an odd set of alerts from a panel on my grafana dashboard. High level notes and more info below:

  • My query aliased “failure” does not have any data
  • My alarm will set state to OK when No Data is available
  • This alarm is alerting multiple times over a 2 day window (despite the settings above)
  • This alert is claiming a query value of “20” for all alerts (not justified in data)

Assuming there is no data for an query, and the alarm allows no data, I would not expect these alerts. And if they were accurate alerts I would expect a variable failure value given variable usage.

Can anyone help explain this behavior?? Any help is appreciated

See screenshots:

image

image
image3250×1416 248 KB

See Test Rule results:

{
   "firing":false,
   "state":"ok",
   "conditionEvals":"false = false",
   "timeMs":"37.435ms",
   "logs":[
      {
         "message":"Condition[0]: Query",
         "data":{
            "from":1671804237126,
            "queries":[
               {
                  "refId":"A",
                  "model":{
                     "datasource":{
                        "type":"graphite",
                        "uid":"000000008"
                     },
                     "hide":false,
                     "refCount":0,
                     "refId":"B",
                     "target":"aliasByNode(stats….failure, 7)",
                     "textEditor":false
                  },
                  "datasource":{
                     "id":8,
                     "name":"graphite"
                  },
                  "maxDataPoints":1500,
                  "intervalMs":200
               }
            ],
            "to":1671804537126
         }
      },
      {
         "message":"Condition[0]: Query Result",
         "data":{
            "fromDataframe":true,
            "series":[
               
            ]
         }
      },
      {
         "message":"Condition: Eval: false, Query Returned No Series (reduced to null/no value)",
         "data":null
      }
   ]
}