I’m using Grafana on a docker swarm (cadvisor/node-exporter/prometheus). All works fine to begin with but after a random amount of time, Grafana dashboard goes blank and red error messages appear all down the screen stating ‘Templating Template variables could not be initialized: undefined’.
Has anybody run into this problem or know what the cause is? I can’t understand why it works perfectly for a random amount of time and then suddenly stops, even in front of my eyes yesterday.
Versions of the software used:
cadvisor:0.36.0
grafana:7.1.5
node-exporter:1.0.1
prometheus:2.19.1
I’m also running under traefik reverse proxy version 2.2.5
edit: I’ve found it’s something to do with the volume that grafana uses in docker - if I remove the volume and deploy the swarm again, there is no problem at connecting to the prometheus instance.
edit 2: I’ve been running a test deploment with 4 nodes for an hour and the error appeared again, but then went away and the deployment has now reset it’s timer and is continuing normally.