MYSQL max's out CPU when Grafana runs a Playlist

We have been using Grafana for a couple years now without any issues. We recently have started using the Playlist feature to rotate through a number of dashboards we have built. Anytime we run a playlist MYSQL’s CPU usage max’s out the server until it crashes. I can’t see why this might be happening. We can use any of the indvidual dashboards without MYSQL using more than baseline CPU. It appears only when playlists are started. Anyone come across this before?

Using Grafana v7.5.3 (3e3cf4dc60)
CPU: 2 vCore
RAM: 4096 MB
Storage:128 GB NVMe

Hi @gregg,

So one quick thought: are the dashboard panels in your playlist loading lots of annotations?

@mattabrams
We don’t have any annotations on our dashboards. The dashboards themselves are very similar but pulling data from different jobs. Not sure if that helps but I can confirm no annotations. Any other ideas? I am happy to provide any information you think would help get to the bottom of this. Thanks @mattabrams

Hi @gregg,

Are you running Grafana and MySQL on the same VM? And I’m assuming that your company is using the core MySQL data source plugin?

Also, can you see a correlation between CPU usage and the speed at which you cycle through the playlist?

Hey @mattabrams,

That is correct Grafana and MySQL are living on the same VPS. We are using the core MYSQL native plugin too.

I haven’t tested to see if CPU usage increses with our playlist cycle speed. I can say we are currently sitting at a playlist that only rotates once per minute across about 8 dashboards. I could push out the rotation rate to 2 minutes and see if that makes any difference to CPU usage. I will test that and let you know.

1 Like

awesome. I’ll do some more investigating on my end as well. Maybe some of the backend folks have encountered this before…

@mattabrams
I have been running the test all day with the cycle rate set to 2mins (double our default), and the CPU usage has gone up to 20% but it has not spiked and killed the server. At this stage I am happy that this solution is working for us.
If anything changes here I will let you know. TBH I didn’t think that was going to work :smile:

1 Like

@mattabrams
I left that test running all night and it seems to have resolved my problem. My refresh on those dashboards must have been taking long enough to cause problems.

Thanks for the help. I will mark this as solved.

@gregg,

first and foremost I’m glad you’ve found a way through this issue. Huzzah!

I’m still intrigued as to what made the faster playlist cycle SO much more resource-intensive. Definitely going to keep asking around about this.

If the issue returns, ping me here and we’ll pick up where we left off! :+1: