Hello,
We are experiencing a performance problem with the new version of grafana (V11.0.0).
The server supporting our mysql database is heavily used by the grafana.exe service (80Mb/s).
When we were on a much older version of grafana (V8), we didn’t have this consumption.
I have the impression that this is due to the fact that grafana loads all the requests at once, because when I open my home dashboard, all my panels load very quickly (or are already loaded). Unlike the old version of grafana, where each panel would load as you scrolled through the dashboard.
Do you know how to remedy this consumption problem? Is it possible to set grafana so that it doesn’t load all requests at once?
Thanks
Enable debug log level (and/or otel tracing) in the Grafana and verify if your impressions is correct and what’s causing that.
My blind guess: alerting with low evaluation periods
1 Like
Hello,
Thanks for your quick reply!
I wasn’t thinking about alerts. Despite the fact that the evaluation time was 10min (which is not so low), it was rather the way I had organized my folders that was the problem. I had around thirty folders, each containing several dozen alerts, and I think that the sequence between all the folders and alerts was therefore not optimal.
After tidying up the architecture of my alert folders, I also followed the recommendation you made on another forum topic, by setting the disable_jitter field to true.
Since then, traffic has dropped considerably! (a few thousand bytes/s).
Many thanks!
1 Like
I come back to my last post, the problem is not solved
I thought the throughput had dropped, but it’s actually cycling, where for 35min I get a decent throughput and then for 25min the throughput increases to 80MB/s and so on (where it was flat at 80MB/s before).
I can’t figure out how to set my alerts and alert groups to avoid this.
In the Grafana documentation, it says: “Grafana-managed alert rules within the same group are evaluated concurrently-they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.”
Does this mean that I should put all my alerts in a single group, so that they run one after the other?
The disable_jitter field doesn’t seem to affect throughput.
I’m happy to guess in this guessing lottery if you provide legal binding with 6+ figures price.
Otherwise don’t guess and follow recommended debugging.
Thank you for your comment.
I did follow your recommendations on debugging (debugg log level), but I confess I don’t know what to learn from it, I can’t figure out what to look for in this log to hope to find the source of the problem.
I’ve reduced the log to just 2 hours, so as not to make it too big. Can you help me understand what I need to look for in this log? What should be obvious ?
Thanks in advance for your help on this subject, which I apparently don’t seem to know very well.
grafana_log.json (7.8 MB)
Look pattern/correlations with that IO utilization metric. Sorry, but don’t expect that I will do that for you. If you are expecting that, then I would recommend to pay for official Grafana support.
1 Like