Synthetic monitoring on local grafana instance

Hello,

i have been installed the grafana synthetic monitoring on my grafana local instance. i could able to see the synthetic monitoring plugin on my local grafana instance. when i am trying to configure the plugin it was keep on loading. i could not find any progress. could you please help me out.

Regards,
Suresh

Hi @sureshpalanisamy, welcome to the community!

Synthetic monitoring runs on your cloud instance, but you can set up the plugin on your local grafana to view it locally. It will still need to be configured to connect to your cloud instance to get the synthetic monitoring data.

Are you following the instructions here for configuring the connection with your cloud instance?

Hi @melori.arellano , Thanks for the update.

No i didn’t create an yaml file. i have just downloaded the plugin and extracted on my grafana local instance. do i need to create the provisioning file under plugin directory?

Regards,
Suresh

@melori.arellano I have created both file under plugin directory of my grafana local instance. still i could see it is keep on loading screen.

Regards,
Suresh

@sureshpalanisamy, I recommend enabling debug logging and restarting grafana. You will be able to see if there are any errors in the plugin configuration.

Enable debug logs

@melori.arellano Hi sorry for the delay. i had some health issue and not able to connect . i have enabled the debug log and i did’nt found any error to troubleshoot. i have created my both files under plugins directory. still no luck.

Regards,
Suresh

@sureshpalanisamy, I hope you’re recovering well. I re-labeled this for grafana cloud.

  • What path is set in grafana.ini for provisioning, verify that it matches where your file is. The line should look like this:

    # folder that contains provisioning config files that grafana will apply
    # on startup and while running.
    
    provisioning = conf/provisioning
    

Hi Suresh,

you need to provision the Synthetic Monitoring plugin and the Grafana Cloud datasources. While you can install the plugin in a local Grafana instance, you need a hosted Grafana stack where you can store the data. At the moment there’s no support for storing the data in self-hosted instances of Prometheus and Loki.

If that’s fine, then you need two files:

/etc/grafana/provisioning/plugins/sm.yaml (name is not important, location is, please check the local Grafana configuration for the correct location)

/etc/grafana/provisioning/datasources/grafana_cloud.yaml (same)

For the datasources it’s something like:

apiVersion: 1

datasources:
  - name: grafanacloud-sm-logs
    type: loki
    access: proxy
    url: ...
    basicAuth: true
    basicAuthUser: ...
    jsonData:
      maxLines: 1000
    secureJsonData:
      basicAuthPassword: ...
    version: 1

  - name: grafanacloud-sm-metrics
    type: prometheus
    access: proxy
    url: ...
    basicAuth: true
    basicAuthUser: ...
    jsonData:
      timeInterval: 1s
    secureJsonData:
      basicAuthPassword: ...
    version: 1

For the plugin:

apiVersion: 1

apps:
  - type: grafana-synthetic-monitoring-app
    name: grafana-synthetic-monitoring-app
    disabled: false
    jsonData:
      apiHost: ...
      stackId: ...
      logs:
        grafanaName: grafanacloud-sm-logs
        hostedId: ...
      metrics:
        grafanaName: grafanacloud-sm-metrics
        hostedId: ...
    secureJsonData:
      publisherToken: ...

The names of the datasources do not matter, but they must be consistent.

The tokens for the datasources are viewer tokens (or editor, for some use cases).

The token for the plugin is a publisher token.

There’s a legacy set up path that requires an admin token, but I would recommend that you try this first.

1 Like

@mem @melori.arellano Thanks a lot for your time.

Let me check the above option and back to you.

Regards,
Suresh

@melori.arellano Please find my location file

Directory where grafana will automatically scan and look for plugins

;plugins = /var/lib/grafana/plugins

folder that contains provisioning config files that grafana will apply on startup and while running.

;provisioning = conf/provisioning

@mem i can go with prometheus and loki for storing

can you help me should i create both yaml under /var/lib/grafana/plugins.

is it possible if we can have screen share session. because i have tried for many times but still no luck.

@sureshpalanisamy I’m reading the docs on github for setting this up and I think we can improve them to make this easier.

Here is what you’ll want to do:

  1. Check that the plugin grafana-synthetic-monitoring-app is installed in your plugins directory /var/lib/grafana/plugins. If it is not, Install the synthetic monitoring plugin on your local instance of grafana into the plugin directory
  2. Configure your local instance of grafana to connect to your cloud prometheus and loki instances so you can view their data in your local instance of grafana.
  • Create an API key with Viewer permissions in the Grafana Cloud portal:

  • Log in to your grafana cloud instance and refer to the grafanacloud-XXXX-prom and grafanacloud-XXXX-logs datasources to get the URL and user name that you will use to configure the datasources locally

  • In your local instance of grafana under Configuration > Datasources > Add Datasource. Create datasource configurations for loki and prometheus to match the ones that exist in grafana cloud. Use your API Viewer key as the password for both.

    Note: The user is the hostedId that you will need to configure the provisioning file. Keep this browser tab open to reference in step 3

  1. Configure the synthetic-monitoring-app using a provisioning file in your conf/provisioning directory.
  • Create an API key with the MetricsPublisher permission
  • Verify that your plugin provisioning file looks like the one @mem shared and the example here.
apiVersion: 1

apps:
  - type: grafana-synthetic-monitoring-app
    name: grafana-synthetic-monitoring-app
    disabled: false
    jsonData:
      apiHost: ...
      stackId: ...
      logs:
        grafanaName: grafanacloud-sm-logs
        hostedId: ...
      metrics:
        grafanaName: grafanacloud-sm-metrics
        hostedId: ...
    secureJsonData:
      publisherToken: ...

stackId is in the URL when you navigate to your stack in grafana.com:

hostedId is the user name that you saw in your loki and prometheus datasources in step 2. They are specific to your prometheus (metrics) and loki (logs) instances.
apiHost URLs are region specific and listed here
publisherToken is the API key with the MetricsPublisher permission
grafanaName is the name of local logs and metrics datasources you created

  1. Restart grafana
1 Like

@melori.arellano i have done all the steps but still the app is not loading on the plugin.

Regards,
Suresh

@sureshpalanisamy with debug logging on you should see a line like this when the plugin loads:

DBUG[03-15|10:19:09] Loading plugin                           logger=plugin.loader path=/usr/local/var/lib/grafana/plugins/grafana-synthetic-monitoring-app/datasource/plugin.json
DBUG[03-15|10:19:09] Loading plugin                           logger=plugin.loader path=/usr/local/var/lib/grafana/plugins/grafana-synthetic-monitoring-app/plugin.json

If there is a configuration problem, then you might see more errors further down or in the browser developer tools. Are you able to see the synthetic monitoring icon
Screen Shot 2022-03-15 at 10.25.53 AM in your local grafana to indicate that the plugin is installed?

You should also see a line where grafana finds your provisioning file:

t=2022-03-15T10:28:58-0600 lvl=dbug msg="Parsing plugin provisioning file" logger=provisioning.plugins path=/Users/meloriarellano/Downloads/grafana-8.3.3/conf/provisioning/plugins file.Name=sm.yaml

1 Like

@melori.arellano hi i have not installed the plugin using grafana cli. i have download the zip file and extracted the on the /var/lib/grafana/plugins directory and on the logs i don’t have any error and plugin was successfully. but once i click on the UI it is keep on loading

@melori.arellano Hi Hope you are doing good. I have done the steps. now i am getting page not found. when i am clicking on the synthetic monitoring app. Could you please help on this.

@sureshpalanisamy I’m not sure why you’d see that error, are you able to load the page properly when you’re using synthetic monitoring in your grafana cloud instance? If there is an issue with it in your instance, you can open a support ticket with grafana cloud support.