Alert on any value of hundreds of values

Grafana Alerting
,
1

Hello,

I am quite new to Grafana, and we are trying to monitor executors’ metrics on tens of instances for each cluster. And we want to alert if the monitored metric of any of the executors’ is above certain threshold.

image
image2444×430 77 KB

  1. I could only see option of alerting upon using an aggregate function like min, max, avg, but what I am looking for is something like any(). I thought a query like
    “max by (cluster_id, instance_id, executor_id) (executor_metrics)” in the panel would work, but even then in the alert tab, I need to specify the aggregate function. Can you please help if there is a direct way or some possible hacks to achieve this?
  2. Since there could be hundreds of values to monitor for an alert, is there an efficient way to handle this scenario?

Thank you.
-rvk.

2

Welcome @rakeshvarma481 to the Grafana forum. It sounds like you are using Prometheus and could use Grafana’s multidimensional alerting. Have you read/watched this?

1 Like
3

Thank you very much @grant2 for your answer. Looks like this is exactly what we are looking for. But is this multidimensional alerting only possible with Grafana version v9? We are using Grafana version v8.4.

However I see that, I could create reduce expression similar to the instructions from the link you shared,

image
image2350×514 49.8 KB

but on the alerting tab when I select query C it shows a warning that “The datasource does not support alerting queries”.
image
image2478×672 42.5 KB

If these expressions can’t be used for alerting, what are they useful for?

Thank you.
-rvk.

4

I think multi-dimensional alerts were there in v8, but they became the default in v9. Can you go ahead and upgrade to v9? Once you do, you should see this on the left:

image

5

@grant2 Unfortunately not. At the moment we are using Amazon Managed Grafana and they are still on version v8.4 at the moment. If you can point me to some documentation/article how this is possible on v8.4 or lesser, that would be of great help.

Or we have to think of switching to custom installation of Grafana on some instance may be.

6

I think in v8.4 you can enable Unified alerting by removing the semicolon in the .ini file (assuming Amazon lets you access this file)

[unified_alerting]
;enabled = true
1 Like
7

@grant2 Thanks very much. No, we don’t have access to the .ini file either.

Is there a possibility that may be you are also aware how to set it up via prometheus alert rules “expr” ?

8

I am not aware of how to set up prometheus alert rules. Sorry.

I think you will be all set once Amazon Managed Grafana joins the rest of us with v.9.

9

@grant2 Got it. Thank you very much for all the answers.