Help building backend datasource (MQTT for streaming data)

I am running 8.1.2 open source Grafana on a fresh install of Debian 11 and have modified the config file per the instructions.

I am trying to install the MQTT datasource plugin. I cloned the plugin and put the extracted folder in my plugins director (first I tried the default folder at /var/lib/grafana/plugins and then created a second folder at /var/lib/grafana/grafana-plugins

First step acc. to these instructions is to change the config file, restart Grafana server, etc. I did all that with no issues.

Then I navigated to the folder created earlier at /var/lib/grafana/grafana-plugins and ran this in terminal:

npx @grafana/toolkit plugin:create my-plugin

and it runs through a bunch of stuff (fetchMetadata, loadDep, checking installable status, etc.) and then after a few minutes says this:

root@Debian-Dell:/var/lib/grafana/grafana-plugins# npx @grafana/toolkit plugin:create my-plugin
internal/modules/cjs/loader.js:892
  throw err;
  ^

Error: Cannot find module '/root/.npm/_npx/7113/lib/node_modules/@grafana/toolkit/node_modules/puppeteer-core/install.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! puppeteer-core@1.18.1 install: `node install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the puppeteer-core@1.18.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-08-20T22_08_30_496Z-debug.log
Install for [ '@grafana/toolkit@latest' ] failed with code 1
root@Debian-Dell:/var/lib/grafana/grafana-plugins# 

And so I never get to the next part of the instruction which is:

Select Backend Datasource Plugin and follow the rest of the steps in the plugin scaffolding command.

Can someone help me with this? I have been Googling for solutions to no avail.

Hey @grant2, I’ll say up front I don’t know the actual answer to your question (i.e. I haven’t built that plugin myself), but I do see an issue.

Unfortunately the readme is not very specific when it says “Refer to: Building a Streaming Datasource Backend Plugin”. I don’t believe you actually need to go through all the steps for building a new plugin that are listed there. Specifically, the npx command that’s erroring out for you is for cloning a skeleton code code base of a sample plugin. I don’t think that’s what you want - you just want to build the code for the MQTT plugin.

I think that after setting up the build pre-requisites (which it sounds like you did), you can go ahead and follow the “Installation steps” for the MQTT plugin. I.e.:

  1. Clone the plugin to your Grafana plugins directory.
  2. Build the plugin by running yarn install and then yarn build.

Maybe give that a try and see what you get?

Thanks Svetb. I proceed with the installation steps and I think I got it set up now. If / when I can get it working, I will document it here for all to see.

Here’s what I now see on the plugins screen (note that I have changed my config file to allow this unsigned plugin)

and on the data sources, I can load the MQTT data source. I enter my MQTT broker’s IP address (10.250) and the port being used (1883) here is what I see. I have no security credentials in use on the broker and I have tested it from two other PCs so I am sure it is correctly relaying the data. When I click Save & Test, I immediately get the “Plugin unavailable” message right away, which tells me it might not be installed correctly.

Yarn install seems to work fine, but yarn build seems to end with an error:

yarn run v1.22.11
$ rm -rf dist && grafana-toolkit plugin:build && mage build:backend
  Using Node.js v14.17.5
  Using @grafana/toolkit v8.0.0-beta.3
✔ Preparing
✔ Linting
ts-jest[config] (WARN) The option `tsConfig` is deprecated and will be removed in ts-jest 27, use `tsconfig` instead
 PASS  src/handleEvent.test.ts

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   2 passed, 2 total
Time:        2.324 s
Ran all test suites with tests matching "".
✔ Running tests
⠋ Compiling...  Starting type checking service...
  Using 1 worker with 2048MB memory limit
⠇ Compiling...  
   Hash: 6b5a018b08c2ac55e195
  Version: webpack 4.41.5
  Time: 4989ms
  Built at: 08/23/2021 5:11:19 PM
                  Asset        Size  Chunks                   Chunk Names
           CHANGELOG.md    53 bytes          [emitted]        
                LICENSE    11.1 KiB          [emitted]        
              README.md    3.69 KiB          [emitted]        
           img/mqtt.svg    1.31 KiB          [emitted]        
              module.js     3.8 KiB       0  [emitted]        module
  module.js.LICENSE.txt   808 bytes          [emitted]        
          module.js.map      24 KiB       0  [emitted] [dev]  module
            plugin.json  1000 bytes          [emitted]        
  Entrypoint module = module.js module.js.map
  [0] external "react" 42 bytes {0} [built]
  [1] external "@grafana/ui" 42 bytes {0} [built]
  [2] external "lodash" 42 bytes {0} [built]
  [3] external "@grafana/data" 42 bytes {0} [built]
  [4] external "@grafana/runtime" 42 bytes {0} [built]
  [5] ./module.ts + 5 modules 14.5 KiB {0} [built]
      | ./module.ts 296 bytes [built]
      | ./datasource.ts 352 bytes [built]
      | ./ConfigEditor.tsx 2.78 KiB [built]
      | ./QueryEditor.tsx 658 bytes [built]
      | ../node_modules/tslib/tslib.es6.js 10 KiB [built]
      | ./handleEvent.ts 395 bytes [built] 
  
✔ Compiling...
/bin/sh: 1: mage: not found
error Command failed with exit code 127.

mage is a pre-requisite for building a datasource backend plugin (see Build a data source backend plugin | Grafana Labs), so you need to install it on your system in order to build.

got same problen when trying to bild plugin

/bin/sh: 1: mage: not found
error Command failed with exit code 127.

Have you fulfilled all steps of the “prerequisites” shown at
Build a data source backend plugin | Grafana Labs ?

Antony.

if i go under /bin into sh it shows

$ mage -version
Mage Build Tool v1.11.0-2-g4cf3cfc
Build Date: 2021-08-29T17:48:31Z
Commit: 4cf3cfc
built with: go1.13.8

In that case, try “which mage” and check whether the directory it’s in is
included in your $PATH.

Antony.

$ which mage
/home/ubuntu/go/bin/mage

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/ubuntu/go/bin:

Okay, I have to defer to someone with more experience building plugins than I,
then, as this looks quite reasonable to me, and I would expect it to work.

Antony.

I installed Go first according to these instructions and I now have a Go folder in the usr/local folder and my version appears as 1.17:

root@Debian-Dell:~# export PATH=$PATH:/usr/local/go/bin
root@Debian-Dell:~# go version
go version go1.17 linux/amd64

Then I installed Mage according to these instructions and have a mage folder in the root directory:
image

And then did the following (which seemed successful since I did not see any error messages):

root@Debian-Dell:~# cd mage
root@Debian-Dell:~/mage# go run bootstrap.go
Running target: Install
exec: go env GOBIN
exec: go env GOPATH
exec: git rev-parse --short HEAD
exec: git describe --tags
exec: go build -o /root/go/bin/mage -ldflags=-X "github.com/magefile/mage/mage.timestamp=2021-08-31T13:22:33-04:00" -X "github.com/magefile/mage/mage.commitHash=4cf3cfc" -X "github.com/magefile/mage/mage.gitTag=v1.11.0-2-g4cf3cfc" github.com/magefile/mage

With Go and Mage apparently installed correctly, I tried yarn install (which gave me no errors, just like last time) and then yarn build, I got a little bit further, but still ended with the /bin/sh: 1: mage: not found

root@Debian-Dell:/var/lib/grafana/grafana-plugins/mqtt-datasource-main# yarn build
yarn run v1.22.11
$ rm -rf dist && grafana-toolkit plugin:build && mage build:backend
  Using Node.js v14.17.5
  Using @grafana/toolkit v8.0.0-beta.3
✔ Preparing
✔ Linting
ts-jest[config] (WARN) The option `tsConfig` is deprecated and will be removed in ts-jest 27, use `tsconfig` instead
 PASS  src/handleEvent.test.ts

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   2 passed, 2 total
Time:        0.713 s, estimated 1 s
Ran all test suites with tests matching "".
✔ Running tests
⠋ Compiling...  Starting type checking service...
  Using 1 worker with 2048MB memory limit
⠦ Compiling...  
   Hash: 6b5a018b08c2ac55e195
  Version: webpack 4.41.5
  Time: 5850ms
  Built at: 08/31/2021 12:39:42 PM
                  Asset        Size  Chunks                   Chunk Names
           CHANGELOG.md    53 bytes          [emitted]        
                LICENSE    11.1 KiB          [emitted]        
              README.md    3.69 KiB          [emitted]        
           img/mqtt.svg    1.31 KiB          [emitted]        
              module.js     3.8 KiB       0  [emitted]        module
  module.js.LICENSE.txt   808 bytes          [emitted]        
          module.js.map      24 KiB       0  [emitted] [dev]  module
            plugin.json  1000 bytes          [emitted]        
  Entrypoint module = module.js module.js.map
  [0] external "react" 42 bytes {0} [built]
  [1] external "@grafana/ui" 42 bytes {0} [built]
  [2] external "lodash" 42 bytes {0} [built]
  [3] external "@grafana/data" 42 bytes {0} [built]
  [4] external "@grafana/runtime" 42 bytes {0} [built]
  [5] ./module.ts + 5 modules 14.5 KiB {0} [built]
      | ./module.ts 296 bytes [built]
      | ./datasource.ts 352 bytes [built]
      | ./ConfigEditor.tsx 2.78 KiB [built]
      | ./QueryEditor.tsx 658 bytes [built]
      | ../node_modules/tslib/tslib.es6.js 10 KiB [built]
      | ./handleEvent.ts 395 bytes [built] 
  
✔ Compiling...
/bin/sh: 1: mage: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@knu33l1983 I’m afraid I’m not sure what’s happening in your case. Might be worth seeing how exactly mage is being called by the yarn build process and troubleshooting that.

@grant2 after you installed mage, did you add it to your path definition? I.e., can you successfully run mage -version from your prompt? [By the way, a point on nomenclature, you say you installed it in your root directory, but I’m pretty sure you mean your home directory. I suspect that’s not best practice, but I’m also not a mage expert]

I’m having some issues also on Ubuntu getting this plug-in to work. I don’t get error messages when running the packages/builds, following the long list of Ubunutu instructrions, however, I can’t seem to locate the plugin when in Grafana desktop :add the source. I posted here, perhaps with some experienced users such as yourself.

Here is my post, with more details:

but I basically ran all the Ubuntu instructions from the comment section and then went to " Verify that the plugin is installed" on the main github page

any help would be appreciated

This topic was automatically closed after 365 days. New replies are no longer allowed.