After provisioning I'm seeing "failed to delete dashboard" errors in the log. How do I make it stop?

hackalot · October 2, 2018, 7:48pm

What Grafana version are you using?

5.2.4

What datasource are you using?

MySQL

What OS are you running grafana on?

Fedora release 28 (Twenty Eight)

What did you do?

I setup Grafana in a Docker container.

sudo docker run -d -p 3000:3000 \
 --name=grafana --restart=always \
 -v /etc/grafana/data:/var/lib/grafana \
 -v /etc/grafana/grafana.ini:/etc/grafana/grafana.ini \
 -v /etc/grafana/provisioning:/etc/grafana/provisioning \
 -e "GF_INSTALL_PLUGINS=grafana-piechart-panel" \
 grafana/grafana

I used the API to setup orgs, users and datasources. I used the ‘folder’ provisioning method to add dashboards to the orgs. All went well and the dashboards show up in the org. Sample YAML:

apiVersion: 1

providers:
- name: 'default'
  orgId: 2
  folder: ''
  type: file
  disableDeletion: false
  updateIntervalSeconds: 60
  options:
    path: /etc/grafana/provisioning/dashboards/2

Now when I check the logs I see that every minute I get a flurry of messages related to failing to delete missing dashboards. A sample:

t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=18
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=18
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=20
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=20
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=28
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=28
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=41
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=41
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=29
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=29
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=11
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=11
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=37
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=45
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=36
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=45
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=3
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=3
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=4
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=36
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=38
t=2018-10-02T18:37:31+0000 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=4
t=2018-10-02T18:37:31+0000 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=7

Thank you in advance for any help you can provide!

hackalot · October 3, 2018, 2:35pm

@torkel

I set up a local grafana and provisioned a couple of dashboards and I’m getting the same messages over there now.

t=2018-10-03T07:32:23-0700 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=5
t=2018-10-03T07:32:23-0700 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=5
t=2018-10-03T07:32:23-0700 lvl=dbug msg="deleting provisioned dashboard. missing on disk" logger=provisioning.dashboard type=file name=default id=6
t=2018-10-03T07:32:23-0700 lvl=eror msg="failed to delete dashboard" logger=provisioning.dashboard type=file name=default id=6

bergquist · October 3, 2018, 3:36pm

Are you by any chance using symlinks?

Seems like Grafana cannot find the dashboard on disk anymore and tries to delete it.
Not sure why. The logstatement for this error lacks information

I’ll create a fix for this and perhaps you can try or nightly build with more logging.

hackalot · October 3, 2018, 6:04pm

@bergquist Thank you for your reply!

Yes, I am… in the remote case I have an EBS volume mounted at /mnt/grafana and have /etc/grafana -> /mnt/grafana.

On my local instance (Mac), I have /usr/local/opt/grafana -> ../Cellar/grafana/5.2.4.

Here is a copy of my local .yaml file:

apiVersion: 1

providers:
- name: 'default'
  orgId: 2
  folder: ''
  type: file
  disableDeletion: false 
  updateIntervalSeconds: 60
  options:
    path: /usr/local/opt/grafana/share/grafana/conf/provisioning/dashboards/2

hackalot · October 4, 2018, 12:19pm

@bergquist Do you have a revision # for the nightly build I should try?

bergquist · October 4, 2018, 12:35pm

https://grafana.com/grafana/download/5.4.0-38047pre1 will include more logging.

hackalot · October 5, 2018, 5:34pm

@bergquist Sorry for my ignorance. Turns out that I have no idea how to install a nightly build on a Mac. I’ve been using brew install grafana to get the lastest stable on there. I just tried brew unlink grafana and then ran brew install --HEAD grafana/grafana/grafana which seems to have pulled down the latest unstable build, which maybe has your changes in it… not sure… but regardless, it displays html/template: "index" is undefined when I navigate to http://grafana.staged-by-discourse.com.

I found this page here:How to install nightly Grafana build or beta build on Mac using Homebrew

But it isn’t step by step instructions and it wasn’t much help.

Can you please explain how I am to install a nightly build on a Mac?

hackalot · October 5, 2018, 5:48pm

Oh, there was a build error…

shaunatdozuki@Shauns-MacBook-Pro ~ $ brew reinstall --HEAD grafana/grafana/grafana
==> Reinstalling grafana/grafana/grafana 
==> Cloning https://github.com/grafana/grafana.git
Updating /Users/shaunatdozuki/Library/Caches/Homebrew/grafana--git
==> Checking out branch master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
HEAD is now at f6d3325 Merge pull request #13529 from grafana/davkal/13517-resize-graph
==> go run build.go build
==> yarn install
==> npm install grunt-cli -ddd --build-from-source --cache=/Users/shaunatdozuki/Library/Caches/Homebrew/npm_cache
==> node_modules/grunt-cli/bin/grunt build --force
Error: An exception occured within a child process:
  Errno::ENOENT: No such file or directory - bin/grafana-cli

shaunatdozuki@Shauns-MacBook-Pro ~ $ ll /usr/local/Cellar/grafana/HEAD-f6d3325/bin/grafana-cli
-r-xr-xr-x  1 shaunatdozuki  wheel    20M Oct  5 10:26 /usr/local/Cellar/grafana/HEAD-f6d3325/bin/grafana-cli

Any idea how to get around this?

hackalot · October 5, 2018, 6:31pm

So I uninstalled everything and started all over. I get through the install of HEAD using brew install --HEAD grafana without errors but I still /usr/local/Cellar/grafana/HEAD-f6d3325/share/grafana/public/views/ still does not have an index.html in it.

Whenever I try to brew install --HEAD grafana/grafana/grafana I get errors about not being able to find bin/grafana-cli.

bergquist · October 8, 2018, 7:50am

I’ve never tried installing HEAD from brew myself. (running linux) But you can always download and run the binaries yourself. https://grafana.com/grafana/download/5.4.0-38558pre1?platform=mac

hackalot · October 8, 2018, 5:35pm

okay… I’ll try on a linux box ASAP.