Couchbase plugin - plugin unavailable error

jkraus0917 · June 24, 2024, 7:11pm

Looking for solution for plugin unavailable error. Installation based on following links:

Environment:
Grafana v11.0.0 (277ef258d4)
(base) # cat /etc/os-release
NAME=“Rocky Linux”
VERSION=“9.4 (Blue Onyx)”
ID=“rocky”
ID_LIKE=“rhel centos fedora”
VERSION_ID=“9.4”

Installed Grafana:
cd /usr/local/src
curl -O --insecure https ://dl.grafana.com/oss/release/grafana-11.0.0-1.x86_64.rpm
yum install grafana-11.0.0-1.x86_64.rpm
vi /etc/grafana/grafana.ini
systemctl start grafana-server

Downloaded and compiled couchbase plugin:
git clone https ://github.com/couchbaselabs/grafana-plugin.git
cd grafana-plugin/
yarn install && yarn build
cd /usr/share/grafana/public/app/plugins/datasource
cp -R /usr/local/src/grafana-plugin/couchbase-datasource/ .
export GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=couchbase-datasource
export GF_PLUGIN_APP_TLS_SKIP_VERIFY_INSECURE=true

NOTE: couchbase-plugin directory does not work in the default plugins directory - needs to be in
/usr/share/grafana/public/app/plugins/datasource to be seen by GUI

vim /etc/grafana/grafana.ini
#[plugins]
allow_loading_unsigned_plugins = couchbase-datasource

systemctl restart grafana-server

Problem:
When adding the new couchbase-datasource and clicking the SAVE & TEST button an error - “Plugin Unavailable” occurs

Logs indicate failed to open plugins path:
/var/log/grafana/grafana.log:

logger=server t=2024-06-24T13:44:19.83677344-05:00 level=info msg="Shutdown started" reason="System signal: terminated"
logger=tracing t=2024-06-24T13:44:19.837055199-05:00 level=info msg="Closing tracing"
logger=plugin.couchbase-datasource t=2024-06-24T13:44:19.837262798-05:00 level=warn msg="plugin failed to exit gracefully"
logger=ticker t=2024-06-24T13:44:19.837443562-05:00 level=info msg=stopped last_tick=2024-06-24T13:44:10-05:00
logger=grafana-apiserver t=2024-06-24T13:44:19.837585711-05:00 level=info msg="StorageObjectCountTracker pruner is exiting"
logger=settings t=2024-06-24T13:44:19.967841499-05:00 level=info msg="Starting Grafana" version=11.0.0 commit=277ef258d4b9a5acdf2932347c6a4ca72d739b28 branch=HEAD compiled=2024-06-24T13:44:19-05:00
logger=settings t=2024-06-24T13:44:19.968337622-05:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2024-06-24T13:44:19.968364767-05:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2024-06-24T13:44:19.968381804-05:00 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2024-06-24T13:44:19.968399339-05:00 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2024-06-24T13:44:19.968409215-05:00 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2024-06-24T13:44:19.968420811-05:00 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2024-06-24T13:44:19.968430774-05:00 level=info msg=Target target=[all]
logger=settings t=2024-06-24T13:44:19.968526017-05:00 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2024-06-24T13:44:19.968540035-05:00 level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2024-06-24T13:44:19.968549113-05:00 level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2024-06-24T13:44:19.968559823-05:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2024-06-24T13:44:19.968568617-05:00 level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2024-06-24T13:44:19.968582292-05:00 level=info msg="App mode production"
logger=sqlstore t=2024-06-24T13:44:19.969549716-05:00 level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2024-06-24T13:44:19.977479442-05:00 level=info msg="Locking database"
logger=migrator t=2024-06-24T13:44:19.977535707-05:00 level=info msg="Starting DB migrations"
logger=migrator t=2024-06-24T13:44:19.992806207-05:00 level=info msg="migrations completed" performed=0 skipped=558 duration=684.572µs
logger=migrator t=2024-06-24T13:44:19.993269395-05:00 level=info msg="Unlocking database"
logger=secrets t=2024-06-24T13:44:19.993966329-05:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=renderer.manager t=2024-06-24T13:44:20.030647214-05:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.angulardetectorsprovider.dynamic t=2024-06-24T13:44:20.031214154-05:00 level=info msg="Restored cache from database" duration=484.663µs
logger=plugin.store t=2024-06-24T13:44:20.032733325-05:00 level=info msg="Loading plugins..."
logger=plugin.sources t=2024-06-24T13:44:20.032780365-05:00 level=error msg="Failed to load external plugins" error="failed to open plugins path"
logger=plugins.registration t=2024-06-24T13:44:20.063086733-05:00 level=error msg="Could not register plugin" pluginId=xychart error="plugin xychart is already registered"
logger=plugins.initialization t=2024-06-24T13:44:20.063127699-05:00 level=error msg="Could not initialize plugin" pluginId=xychart error="plugin xychart is already registered"
logger=plugin.store t=2024-06-24T13:44:20.063293937-05:00 level=info msg="Plugins loaded" count=54 duration=30.561601ms
logger=query_data t=2024-06-24T13:44:20.066929059-05:00 level=info msg="Query Service initialization"
logger=live.push_http t=2024-06-24T13:44:20.069819203-05:00 level=info msg="Live Push Gateway initialization"
logger=ngalert.notifier.alertmanager org=1 t=2024-06-24T13:44:20.072678581-05:00 level=info msg="Applying new configuration to Alertmanager" configHash=a013a3f424edb13bed8050eaf374d506
logger=ngalert.state.manager t=2024-06-24T13:44:20.079095776-05:00 level=info msg="Running in alternative execution of Error/NoData mode"
logger=infra.usagestats.collector t=2024-06-24T13:44:20.080801379-05:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=server t=2024-06-24T13:44:20.080950745-05:00 level=info msg="Writing PID file" path=/var/run/grafana/grafana-server.pid pid=1409856
logger=provisioning.alerting t=2024-06-24T13:44:20.081684183-05:00 level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2024-06-24T13:44:20.081720056-05:00 level=info msg="finished to provision alerting"
logger=renderer.manager t=2024-06-24T13:44:20.08192313-05:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=http.server t=2024-06-24T13:44:20.085050503-05:00 level=info msg="HTTP Server Listen" address=[::]:8898 protocol=http subUrl= socket=
logger=ngalert.state.manager t=2024-06-24T13:44:20.085225429-05:00 level=info msg="Warming state cache for startup"
logger=grafanaStorageLogger t=2024-06-24T13:44:20.090370318-05:00 level=info msg="Storage starting"
logger=provisioning.dashboard t=2024-06-24T13:44:20.09135504-05:00 level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2024-06-24T13:44:20.091384702-05:00 level=info msg="finished to provision dashboards"
logger=ngalert.multiorg.alertmanager t=2024-06-24T13:44:20.093704437-05:00 level=info msg="Starting MultiOrg Alertmanager"
logger=ngalert.state.manager t=2024-06-24T13:44:20.104866264-05:00 level=info msg="State cache has been initialized" states=0 duration=19.640066ms
logger=ngalert.scheduler t=2024-06-24T13:44:20.104990734-05:00 level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1
logger=ticker t=2024-06-24T13:44:20.105094749-05:00 level=info msg=starting first_tick=2024-06-24T13:44:30-05:00
logger=grafana.update.checker t=2024-06-24T13:44:20.150173898-05:00 level=info msg="Update check succeeded" duration=58.756247ms
logger=plugins.update.checker t=2024-06-24T13:44:20.150391708-05:00 level=info msg="Update check succeeded" duration=58.4646ms
logger=context userId=1 orgId=1 uname=admin t=2024-06-24T13:44:20.446181923-05:00 level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=10.100.1.48 time_ms=3 duration=3.521511ms size=0 referer= handler=/api/live/ws status_source=server
logger=grafana-apiserver t=2024-06-24T13:44:20.487161506-05:00 level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-06-24T13:44:20.487828502-05:00 level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
logger=context userId=1 orgId=1 uname=admin t=2024-06-24T13:44:21.791846402-05:00 level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=10.100.1.48 time_ms=5 duration=5.764534ms size=0 referer= handler=/api/live/ws status_source=server

jangaraj · June 24, 2024, 7:32pm

Why you are not using this plugin path, reported by Grafana?

jkraus0917 · June 24, 2024, 7:41pm

It appears the default installation is this path:
/usr/share/grafana/public/app/plugins/datasource

I cannot see my plugin even show up in the GUI without moving it there.

The /var/lib/grafana path does not have a plugins directory:
(base) # ls -l /var/lib/grafana/
total 980
drwx------. 2 grafana grafana 6 Jun 24 12:33 csv
-rw-r-----. 1 grafana grafana 1003520 Jun 24 14:36 grafana.db
drwx------. 2 grafana grafana 6 Jun 24 12:33 pdf
drwx------. 2 grafana grafana 6 Jun 24 12:33 png

I can update the grafana.ini or make a plugins directory in /var/lib/grafana/ and try moving the couchbase-connector there

jangaraj · June 24, 2024, 7:47pm

Please format logs and config snippets.

That’s exactly what your logs are saying:

"Failed to load external plugins" error="failed to open plugins path"

Don’t wait and just make that folder structure for plugins because your Grafana expects it there. Logs are meaningful in this case.

jkraus0917 · June 24, 2024, 7:53pm

Note sure why the base install did not create the directory that the config is pointing to, but I went ahead and added the directory with correct permissions and now at least the logs are showing it is attempting to load the plugin, but now there are file related errors:

logger=plugin.signature.validator t=2024-06-24T14:47:46.728261478-05:00 level=warn msg="Permitting unsigned plugin. This is not recommended" pluginId=couchbase-datasource
logger=plugins.registration t=2024-06-24T14:47:46.729043945-05:00 level=info msg="Plugin registered" pluginId=couchbase-datasource
logger=plugins.backend.start t=2024-06-24T14:47:46.729410437-05:00 level=error msg="Could not start plugin" pluginId=couchbase-datasource error="fork/exec /var/lib/grafana/plugins/couchbase-datasource/dist/gpx_couchbase_linux_amd64: no such file or directory"
logger=plugins.initialization t=2024-06-24T14:47:46.729445383-05:00 level=error msg="Could not initialize plugin" pluginId=couchbase-datasource error="fork/exec /var/lib/grafana/plugins/couchbase-datasource/dist/gpx_couchbase_linux_amd64: no such file or directory"

Not sure why there are files not in the correct directory for the plugin as I did not get any errors compiling it.

(base) [root@r-server2 dist]# pwd
/var/lib/grafana/plugins/couchbase-datasource/dist
(base) [root@r-server2 dist]# ls -l
total 40
-rw-r--r--. 1 grafana grafana    53 Jun 24 12:40 CHANGELOG.md
drwxr-xr-x. 2 grafana grafana    44 Jun 24 12:40 img
-rw-r--r--. 1 grafana grafana  1094 Jun 24 12:40 LICENSE
-rw-r--r--. 1 grafana grafana  5083 Jun 24 12:40 module.js
-rw-r--r--. 1 grafana grafana 13064 Jun 24 12:40 module.js.map
-rw-r--r--. 1 grafana grafana  1193 Jun 24 12:40 plugin.json
-rw-r--r--. 1 grafana grafana   701 Jun 24 12:40 README.md

jkraus0917 · June 24, 2024, 7:58pm

Output from compiling the plugin as per the git link:

The result seems to not have the required file in the ./couchbase-datasource/dist directory

(base) [root@r-server2 couchbase-datasource]# yarn install & yarn build
[1] 1410979
yarn run v1.22.22
yarn install v1.22.22
$ webpack -c ./.config/webpack/webpack.config.ts --env production
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@^7.5.1"
success Already up-to-date.
Done in 0.58s.
assets by path img/*.png 126 KiB
  asset img/screenshot.png 100 KiB [compared for emit] [from: img/screenshot.png] [copied]
  asset img/logo.png 25.4 KiB [compared for emit] [from: img/logo.png] [copied]
assets by path *.md 754 bytes
  asset README.md 701 bytes [compared for emit] [from: ../README.md] [copied]
  asset CHANGELOG.md 53 bytes [compared for emit] [from: ../CHANGELOG.md] [copied]
asset module.js 4.96 KiB [compared for emit] [minimized] (name: module) 1 related asset
asset plugin.json 1.17 KiB [emitted] [from: plugin.json] [copied]
asset LICENSE 1.07 KiB [compared for emit] [from: ../LICENSE] [copied]
cached modules 18 KiB (javascript) 937 bytes (runtime) [cached] 14 modules
webpack 5.91.0 compiled successfully in 4720 ms
Done in 6.56s.
[1]+  Done                    yarn install
(base) [root@r-server2 couchbase-datasource]# ls -l
total 588
-rw-r--r--.   1 root root     53 Jun 24 12:34 CHANGELOG.md
drwxr-xr-x.   3 root root    126 Jun 24 12:35 dist
-rw-r--r--.   1 root root    424 Jun 24 12:34 docker-compose.yaml
-rw-r--r--.   1 root root   5061 Jun 24 12:34 go.mod
-rw-r--r--.   1 root root  35928 Jun 24 12:34 go.sum
-rw-r--r--.   1 root root    281 Jun 24 12:34 jest.config.js
-rw-r--r--.   1 root root     77 Jun 24 12:34 jest-setup.js
-rw-r--r--.   1 root root   1094 Jun 24 12:34 LICENSE
-rw-r--r--.   1 root root    324 Jun 24 12:34 Magefile.go
drwxr-xr-x. 903 root root  28672 Jun 24 12:35 node_modules
-rw-r--r--.   1 root root   2508 Jun 24 12:34 package.json
drwxr-xr-x.   3 root root     35 Jun 24 12:34 pkg
-rw-r--r--.   1 root root    701 Jun 24 12:34 README.md
drwxr-xr-x.   3 root root    137 Jun 24 12:34 src
-rw-r--r--.   1 root root     43 Jun 24 12:34 tsconfig.json
-rw-r--r--.   1 root root 472395 Jun 24 12:34 yarn.lock
(base) [root@r-server2 couchbase-datasource]# ls -l ./dist
total 40
-rw-r--r--. 1 root root    53 Jun 24 12:35 CHANGELOG.md
drwxr-xr-x. 2 root root    44 Jun 24 12:35 img
-rw-r--r--. 1 root root  1094 Jun 24 12:35 LICENSE
-rw-r--r--. 1 root root  5083 Jun 24 12:35 module.js
-rw-r--r--. 1 root root 13064 Jun 24 12:35 module.js.map
-rw-r--r--. 1 root root  1193 Jun 24 14:55 plugin.json
-rw-r--r--. 1 root root   701 Jun 24 14:55 README.md
(base) [root@r-server2 couchbase-datasource]#

jangaraj · June 24, 2024, 8:06pm

You are missing “backend”. Just download it from latest Github release (or hack it and compile it on your own, see grafana-plugin/.github/workflows/release.yml at main · couchbaselabs/grafana-plugin · GitHub there is example how author is doing that)

jkraus0917 · June 24, 2024, 8:46pm

Thanks, let me chase down the couchbase plugin developer community. I can at least verify the plugin is now trying to load.

yosiasz · June 24, 2024, 8:54pm

does cb provide a rest api access?

jkraus0917 · June 24, 2024, 9:09pm

I believe so, but we have not used it as of yet. I think the git source distribution is missing files. If I download the .zip for windows I can see that dist folder contains the missing gpx files and if I manually copy those into the /var/lib/grafana/plugins/couchbase-datasource/dist directory I can get the plugin to pass the TEST & SAVE without the error. So I think it is fixed now. Thanks

(base) [root@r-server2 dist]# chmod 755 gpx_couchbase*
(base) [root@r-server2 dist]# ls -l
total 98232
-rw-r--r--. 1 grafana grafana       53 Jun 24 12:40 CHANGELOG.md
-rwxr-xr-x. 1 grafana grafana 18070128 Jun 24 16:02 gpx_couchbase_darwin_amd64
-rwxr-xr-x. 1 grafana grafana 18009090 Jun 24 16:02 gpx_couchbase_darwin_arm64
-rwxr-xr-x. 1 grafana grafana 16293888 Jun 24 16:02 gpx_couchbase_linux_amd64
-rwxr-xr-x. 1 grafana grafana 15597568 Jun 24 16:02 gpx_couchbase_linux_arm
-rwxr-xr-x. 1 grafana grafana 15728640 Jun 24 16:02 gpx_couchbase_linux_arm64
-rwxr-xr-x. 1 grafana grafana 16846336 Jun 24 16:02 gpx_couchbase_windows_amd64.exe
drwxr-xr-x. 2 grafana grafana       44 Jun 24 12:40 img
-rw-r--r--. 1 grafana grafana     1094 Jun 24 12:40 LICENSE
-rw-r--r--. 1 grafana grafana     5083 Jun 24 12:40 module.js
-rw-r--r--. 1 grafana grafana    13064 Jun 24 12:40 module.js.map
-rw-r--r--. 1 grafana grafana     1193 Jun 24 12:40 plugin.json
-rw-r--r--. 1 grafana grafana      701 Jun 24 12:40 README.md

Grafana logs:

logger=plugin.signature.validator t=2024-06-24T16:04:40.116683333-05:00 level=warn msg="Permitting unsigned plugin. This is not recommended" pluginId=couchbase-datasource
logger=plugins.registration t=2024-06-24T16:04:40.116882691-05:00 level=info msg="Plugin registered" pluginId=couchbase-datasource

Previous errors:

logger=plugin.signature.validator t=2024-06-24T14:51:53.1465639-05:00 level=warn msg="Permitting unsigned plugin. This is not recommended" pluginId=couchbase-datasource
logger=plugins.registration t=2024-06-24T14:51:53.146842776-05:00 level=info msg="Plugin registered" pluginId=couchbase-datasource
logger=plugins.backend.start t=2024-06-24T14:51:53.147268135-05:00 level=error msg="Could not start plugin" pluginId=couchbase-datasource error="fork/exec /var/lib/grafana/plugins/couchbase-datasource/dist/gpx_couchbase_linux_amd64: no such file or directory"
logger=plugins.initialization t=2024-06-24T14:51:53.147303032-05:00 level=error msg="Could not initialize plugin" pluginId=couchbase-datasource error="fork/exec /var/lib/grafana/plugins/couchbase-datasource/dist/gpx_couchbase_linux_amd64: no such file or directory"

yosiasz · June 24, 2024, 9:18pm

ok now for the real test no errors but does it work in a dashboard?

jangaraj · June 24, 2024, 9:22pm

No, git contains source code, which must be compiled - you did only frontend part, but not backend (golang code, compiled into those gpx binaries).

jkraus0917 · June 24, 2024, 9:38pm

We’ll start testing tomorrow and see if copying the gpx from the ZIP linked at https://github.com/couchbaselabs/grafana-plugin/releases works, but the installation documentation is not clear about additional steps for compiling the backend gpx files - it seems to imply the yarn install & yarn build should just work which for me does not. The only hint I see are some warnings, but it still seems to complete without an error:

warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@7.8.1"
warning Resolution field "rxjs@7.3.0" is incompatible with requested version "rxjs@^7.5.1"

jkraus0917 · June 24, 2024, 9:57pm

Unfortunately it appears the hacked version of gpx files being copied from the ZIP does not work as the queries do make it to our couchbase environment but return various errors such as this:
logger=context userId=1 orgId=1 uname=admin t=2024-06-24T16:54:09.304292129-05:00 level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=10.100.1.48 traceID=

And Grafana reports as a plugin error. So I’ll have to figure out how to get those gpx binaries to compile to see if that fixes this issue.

yosiasz · June 24, 2024, 11:04pm

easy peasy with infinity rest api plugin (an official grafana provided plugin)

data from couchbase

jkraus0917 · June 25, 2024, 3:50pm

The instructions for the plugin only mention to execute:
yarn install & yarn build
There are no additional “backend” tasks mentioned. I am not a developer and just assume the link you provided to the release.yml file which includes:

      - name: Build backend
        working-directory: couchbase-datasource
        if: steps.check-for-backend.outputs.has-backend == 'true'
        uses: magefile/mage-action@v1
        with:
          version: latest
          args: buildAll

Implies the Magefile.go file will be executed which seems to have a broken URL:
build "github.com/grafana/grafana-plugin-sdk-go/build"

That URL responds with 404 NOT FOUND, however if I update the URL to:
build "github.com/grafana/grafana-plugin-sdk-go/tree/main/build"
It appears to work. but still the gpx files still are not built and stored in the ./couchbase-datasource/dist so I am lost on what to do. It seems the gpx_couchbase_linux_amd64 file is not provided in the installation process

dmitriichechetkin · July 12, 2024, 4:34pm

Hi,
Grafana changed build process last year and, while I updated the ./run.sh script with new build commands, I never updated the readme with them. I’ve fixed that today, apologies for the confusion. Here’s the full command: yarn install && yarn build && mage -v.

Topic		Replies	Views
There is a [Plugin unavailable] error when i add iotdb datasource plugin Explore plugins	1	2487	May 2, 2023
Plugin not found, no installed plugin with that id Grafana	2	11424	February 28, 2020
ChunkLoadError (Influxdb Plugin Failed) Configuration influxdb	17	345	June 23, 2024
CSV datasource when save&test ed shows plugin unavailable error Dashboards csv	0	486	September 4, 2023
FlightSQL Data Source Plug-in Not Working Grafana Cloud plugins , datasource , community-plugins	5	710	May 25, 2023

Couchbase plugin - plugin unavailable error

Related topics