Grafana "Templating Template variables could not be initialized: undefined" error in docker swarm

I’m using Grafana on a docker swarm (cadvisor/node-exporter/prometheus). All works fine to begin with but after a random amount of time, Grafana dashboard goes blank and red error messages appear all down the screen stating ‘Templating Template variables could not be initialized: undefined’.

Has anybody run into this problem or know what the cause is? I can’t understand why it works perfectly for a random amount of time and then suddenly stops, even in front of my eyes yesterday.

Versions of the software used:


I’m also running under traefik reverse proxy version 2.2.5

edit: I’ve found it’s something to do with the volume that grafana uses in docker - if I remove the volume and deploy the swarm again, there is no problem at connecting to the prometheus instance.

edit 2: I’ve been running a test deploment with 4 nodes for an hour and the error appeared again, but then went away and the deployment has now reset it’s timer and is continuing normally.