Issues with invisible but installed plugins

Grafana
1

Hi Grafana Community,

First post as a complete newbie who is trying to have Grafana running on localhost, in order to learn how I can setup a BI dashboard for my company, inspired by latest GrafanaCon video: https://youtu.be/-xlchgoqkqY

I installed grafana through Brew on my macos. I can access Grafana through community.grafana.com and have a first MySQL panel running. Great so far!
When I add new plugins, I can’t seem to have them visible in the Grafana UI. They are however installed under the usr<U+2069> ▸ <U+2068>local<U+2069> ▸ <U+2068>var<U+2069> ▸ <U+2068>lib<U+2069> ▸ <U+2068>grafana<U+2069> ▸ <U+2068>plugins<U+2069> directory.

27
271724×1238 400 KB

Digging deeper, I came across several posts here on the community board and I started looking into the grafana.log file. It shows some weird port:3000 issues:

t=2019-04-13T23:34:57+0200 lvl=info msg=“Starting Grafana” logger=server version=6.1.0 commit=unknown-dev branch=master compiled=2019-04-05T02:22:26+0200
t=2019-04-13T23:34:57+0200 lvl=info msg=“Config loaded from” logger=settings file=/usr/local/opt/grafana/share/grafana/conf/defaults.ini
t=2019-04-13T23:34:57+0200 lvl=info msg=“Config loaded from” logger=settings file=/usr/local/etc/grafana/grafana.ini
t=2019-04-13T23:34:57+0200 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/usr/local/var/lib/grafana”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/usr/local/var/log/grafana”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/usr/local/var/lib/grafana/plugins”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Path Home” logger=settings path=/usr/local/opt/grafana/share/grafana
t=2019-04-13T23:34:57+0200 lvl=info msg=“Path Data” logger=settings path=/usr/local/var/lib/grafana
t=2019-04-13T23:34:57+0200 lvl=info msg=“Path Logs” logger=settings path=/usr/local/var/log/grafana
t=2019-04-13T23:34:57+0200 lvl=info msg=“Path Plugins” logger=settings path=/usr/local/var/lib/grafana/plugins
t=2019-04-13T23:34:57+0200 lvl=info msg=“Path Provisioning” logger=settings path=/usr/local/opt/grafana/share/grafana/conf/provisioning
t=2019-04-13T23:34:57+0200 lvl=info msg=“App mode production” logger=settings
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing HTTPServer” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing SqlStore” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2019-04-13T23:34:57+0200 lvl=info msg=“Starting DB migration” logger=migrator
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing InternalMetricsService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing SearchService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing PluginManager” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Starting plugin search” logger=plugins
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=FlowCharting
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=“Worldmap Panel”
t=2019-04-13T23:34:57+0200 lvl=eror msg=“Plugins: Failed to load plugin json file: /usr/local/var/lib/grafana/plugins/raintank-worldping-app/dist/grafana-worldmap-panel/plugin.json, err: Plugin with same id already exists”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=“worldPing CTA”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=“worldPing Endpoint List”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=“worldPing Endpoint Nav”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Registering plugin” logger=plugins name=worldPing
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing RenderingService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing AlertingService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing DatasourceCacheService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing HooksService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing LoginService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing QuotaService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing RemoteCache” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing ServerLockService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing TracingService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing UsageStatsService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing UserAuthTokenService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing CleanUpService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing NotificationService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing ProvisioningService” logger=server
t=2019-04-13T23:34:57+0200 lvl=info msg=“Initializing Stream Manager”
t=2019-04-13T23:34:57+0200 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2019-04-13T23:34:57+0200 lvl=eror msg=“Stopped HTTPServer” logger=server reason=“listen tcp 0.0.0.0:3000: bind: address already in use”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped Stream Manager”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped RenderingService” logger=server reason=nil
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped NotificationService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped UserAuthTokenService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped ProvisioningService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped CleanUpService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped TracingService” logger=server reason=nil
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped InternalMetricsService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped RemoteCache” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped AlertingService” logger=server reason=“context canceled”
t=2019-04-13T23:34:57+0200 lvl=info msg=“Stopped UsageStatsService” logger=server reason=“context canceled”
t=2019-04-13T23:34:58+0200 lvl=info msg=“Stopped PluginManager” logger=server reason=“context canceled”
t=2019-04-13T23:34:58+0200 lvl=eror msg=“Server shutdown” logger=server reason=“listen tcp 0.0.0.0:3000: bind: address already in use”

Finally, I did try to kill any process that might be already using the 3000 port, but this doesn’t seem to solve anything. Given my newbie level of experience, the following observation might be ridiculous:
When I stop the grafana service in Brew (brew services stop grafana), I’m still able to load & see the community.grafana.com grafana dashboard in my browser (even after a cmd+shift+r). How is this possible?

Thanks for your help!

2

Ok, things are getting interesting:

I have been mucking around with Grafana in the past, so I was thinking that there might be another Grafana server running somewhere, occupying the 3000 port. I changed the port to 3001 in the default.ini file (usr/local/Cellar/grafana/6.1.0/share/grafana/conf/).

Now I have a Grafana UI visible at both community.grafana.com and localhost:3001. The 3001 does show the latest installed plugins, so I have now found my actual, most recent Grafana instance.

The question now is: how can I find the original, older Grafana instance that is still running through port 3000? Any system file searches through Finder for grafana-server only point to usr⁩ ▸ ⁨local⁩ ▸ ⁨Cellar⁩ ▸ ⁨grafana⁩ ▸ ⁨6.1.0⁩ ▸ ⁨bin⁩

3

On a linux machine you could run
ps -aux |grep grafana
to find running processes, I don’t know what the equivalent is on a Mac though.
It may be you are running the same executable twice with different ini files. Again I don’t know how processes get automatically started on boot on a mac but presumably that is the mechanism that is starting the other process (I presume you have rebooted).
Also searching whatever is the equivalent of syslog following a reboot might show how it is starting.

4

Thanks for the suggestion, I found the solution:

  1. execute sudo lsof -i tcp:3000 in terminal
  2. note the PID of the running process, “grafana-s” with PID 38045 in this case
  3. open Activity Monitor on mac and find that PID back
  4. terminate that specific PID
  5. open defaults.ini and change http port back to 3000 and save
  6. restart grafana in terminal: brew services restart grafana
5

How did the extra one get started in the first place? If you don’t know then it will surely happen again.

6

I might have started it manually or through some other ways. It’s been quite a journey full of unknown terminal commands that I executed without much knowledge or experience. Perhaps I started an older Grafana service without knowing. I’ll keep an eye on it, but so far, the grafana.log looks healthy!

7

In that case a reboot would have shut it down, I would have thought.