How to create a table to show results when X number of values exceed threshold

Grafana Alerting
,
1

I’m new to Grafana and I’m not sure if the proper “alerting” feature is needed. What I’m trying to do is a query that shows a table that only includes results in the table when the same nodeHostname is above the threshold (say 1) greater than or equal to 2 times.

My time series is as follows

EM_lastResponseTime{accountId="1", nodeHostname="Node1", site="siteA"} 0.160
EM_lastResponseTime{accountId="1", nodeHostname="Node1" site="siteA"} 3
EM_lastResponseTime{accountId="1", nodeHostname="Node1" site="siteA"} 2
EM_lastResponseTime{accountId="1", nodeHostname="Node1" site="siteA"} .1

EM_lastResponseTime{accountId="1", nodeHostname="Node2", site="siteA"} 0.160
EM_lastResponseTime{accountId="1", nodeHostname="Node2", site="siteA"} .2
EM_lastResponseTime{accountId="1", nodeHostname="Node2", site="siteA"} 3
EM_lastResponseTime{accountId="1", nodeHostname="Node2", site="siteA"} .1

EM_lastResponseTime{accountId="2", nodeHostname="Node3", site="siteA"} 2
EM_lastResponseTime{accountId="2", nodeHostname="Node3", site="siteA"} 3
EM_lastResponseTime{accountId="2", nodeHostname="Node3", site="siteA"} .1
EM_lastResponseTime{accountId="2", nodeHostname="Node3", site="siteA"} .4

I would want my table to include only Node1 and Node3 since in the time window I specified those have >= 2 results that are over 1. I do not want to include Node2 as the one result would be a ‘spike’.

I can’t simply do my query EM_lastResponseTime > 1 as that would include Node2. I need to do some kind of grouping operation I believe.

|  accountId              |       time              |       nodeHostname                          |    Value
|           1             |    XXXXX                  |     Node1                                   |              3
|           1             |    XXXXX                  |     Node1                                   |              2
|           2             |    XXXXX                  |     Node3                                   |              2
|           2             |    XXXXX                  |     Node3                                   |              3

I can live with the above but a bonus table would be simply showing one line for entire ACCOUNTS that have any node exceeding the threshold > X times eliminating the need to show each node. Basically I’d only show Account 1 and Account 2 in the table.

|  accountId             |    Value
|           1            |      ANY
|           2            |      ANY

My data source is Prometheus.

The goal is to have a team investigate these ‘suspect’ accounts (accounts where nodes exceed a threshold more than N times). I’ve validated my data correlates w\ issues.