Need to configure 30000 Grafana Alerts

Hello Everyone,

I have to configure around 30000 alert rules for OPC tags in Grafana, what would be the best way to do it? I am currently using InfluxDB as the data source.

I tried to create around 3000 alert rules using Grafana-managed alerts, but it has lots of performance issues, Grafana started to freeze and became very unresponsive.

Kindly suggest how to overcome this problem.

Thank you.

@aceankit99 have you had a chance to review the suggestions for alerting here:

Performance is a complex puzzle. Do you have any other context to share with the community about what you’ve observed with resource utilization?

  • What does CPU, RAM, & disk usage look like when your instance freezes up?
  • Are there any errors in your Grafana or InfluxDB logs when it freezes?
  • Which database are you using? Is it local to your Grafana instance? Do you see any indication that the database or its’ resources could be a bottleneck?
  • What is the frequency that your alerts are evaluating (5m, 10m, etc)? Are they all evaluating at the same time or is it staggered?
  • Are all of the alerts querying the same Influx datasource? Does the datasource itself show any performance concerns?