Is this possible to build a backend plugin that send front and back HTTP requests?

Grafana Plugin Development
, ,
1

Hello,

I build a HTTP backend datasource plugin from this documentation that have to send HTTP requests to several servers.

  1. Get business value from an API
  2. Get data value from Elasticsearch database

I have a previous HTTP datasource plugin that makes both but it’s now obsolete (HTTP request to Elasticsearch triggers CORS issues).

I added the previous frontend part in the new backend plugin.

I can send HTTP request from the frontend part of the plugin into my business API.
I can send HTTP request from the frontend part into Elasticsearch.

I would like to send those requests into Elasticsearch from the backend part of the plugin.

I can now see data from Elasticsearch with the datasource.ts file; How can I switch the app to use the datasource.go file ?

Is it possible to send requests to my business API from the front part and to my Elasticsearch database from the backend part ?

PS : I noticed that the “Save and test” button in the ConfigEditor runs the datasource.ts instead of the datasource.go file.

Thank you

2

@maximedousselaere To prevent CORS issues, you can use the Proxy mode which going thru backend. You don’t need to develop a backend for that, if I understand the question.

1 Like
3

Hello,

Thank you for your answer.

Could you explain me how to enabled the proxy-mode ?

I don’t have access to the configuration of Grafana.
Thank you

4

@maximedousselaere Check the Add a proxy route section:

5

Hello,

I followed the steps and still have CORS issues. (Route > Datasource ctor and query method)

Actually, when I send request to an Elastic database v7.17.12 it works but if I send request to an Elastic database v8.9.1 I have CORS issues.

Here is what my browser display for Elastic v8.9.1 :

grafanaelkcorsissue
grafanaelkcorsissue786×288 18.5 KB

I have to upgrade all the Elastic databases into version 8.9.1.
All of the requests sent by the datasource plugin are made from the front (Business API & Elasticsearch database). I’ve read that making these by the back part would avoid CORS issues. This is why I’ve build a backend plugin.

What could I do ?

Thank you

6

Hi @maximedousselaere if you are getting cors issues is because you are trying to fetch data directly from the elastic api instead of using the grafana backend proxy service.

Take a closer read at this Add authentication for data source plugins | Grafana Plugin Tools

you should define a route and then use it in your frontend by calling the getBackendSrv() with the url passed to the panel, if you call elastic directly you willl get cors issues.

7

Thank you for your anwser.
I don’t really understand which URL have to be in the routes object.
Is this the URL to Elasticsearch ? Or the URL to my Proxy ? Another URL ?

Same in the getBackendSrv() method, can you confirm it is :

const response = await getBackendSrv()
        .datasourceRequest({
          url,
          method: 'get',
          headers: this.getESAuthorizationHeaders(),
        })

With this.url = instanceSettings.url;

Thank you in advance

8

Hi @maximedousselaere. There are generally two ways you can do about this.

If your plugin defines your endpoint in settings.url[1] then you can directly call instanceSettings.url in the same fashion you are showing me in your code.

The second way you can do this is via routes:

in your plugin.json you can define a route as this:

  "routes": [
    {
      "path": "api",
      "url": "{{ .JsonData.elasticUrl }}"
    }
  ],

this will require that JsonData.elasticUrl to be populated with the correct url via your ConfigEditor.

then in your datasource code you can query this like this:

...
    const endpoint = "/someElasticEndpoint"
    const response = getBackendSrv().fetch({
      // baseUrl is instanceSettings.url
      // notice how I am using `api` which is the path 
      // I defined in routes.
      url: `${this.baseUrl}/api/${endpoint}`, 
    });
    const responseData = await lastValueFrom(response);
    console.log(responseData); // just for example
...

[1] what is settings.url? settings is the place where all the plugin settings live. jsonData is an object inside those settings. somewhere in your plugin config editor you do something like this:

    onOptionsChange({ ...options, jsonData:{
      ...options.jsonData,
      myField: event.target.value
    } });

that is modifying settings. you can set settings.url like this in your Configuration component:

    onOptionsChange({ ...options, 
      url: newUrlValueMaybeFromInput
    });

To ease this work and avoid confusions you can use the DataSourceHttpSettings to set this field easily for you with a lot of different options regarding the http request your user can setup.

It is important you don’t confuse settings.url with for example, settings.jsonData.url. the later is a custom field just as any other you can create inside jsonData.

1 Like
9

Hello,

I’ve added the route elastic as shown :

"routes" : [
    {
      "path" : "elastic",
      "url" : "{{ .JsonData.elasticSearchPath }}",
      "method" : "POST",
      "headers" : [
        {
          "name" : "Accept",
          "content" : "application/json"
        },
        {
          "name" : "Content-Type",
          "content" : "application/json"
        }
      ]
    }
  ]

Then send my request in datasource.ts by :

const fullURLbyProxy = this.url + "/elastic/";
console.log("🧀 fullURLbyProxy : ", fullURLbyProxy)
const response = await getBackendSrv()
      .fetch({
        url: fullURLbyProxy,
        method: 'POST',
        data: postData,
        headers: this.getESAuthorizationHeaders(),
      })

This is the result :

withproxy
withproxy967×172 18.3 KB

I think the fullURLbyProxy has the right value.

How can I confirm that .JsonData.elasticSearchPath has the good value ? Where can I display it ?

If it can help, here are the docker logs :

logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/f1555d2f-9718-43a1-85b4-9d9b0a0136f2/elastic/ remote_addr=172.20.0.1 referer="http://localhost:3000/d/abc5e5d5-46ae-44c7-b78b-aff6188e6804/new-dashboard?orgId=1" t=2024-01-24T15:01:13.956532809Z level=error msg="Proxy request failed" err="Bad Gateway"
logger=context userId=1 orgId=1 uname=admin t=2024-01-24T15:01:13.956653841Z level=error msg="Request Completed" method=POST path=/api/datasources/proxy/uid/f1555d2f-9718-43a1-85b4-9d9b0a0136f2/elastic/ status=502 remote_addr=172.20.0.1 time_ms=71 duration=71.001708ms size=0 referer="http://localhost:3000/d/abc5e5d5-46ae-44c7-b78b-aff6188e6804/new-dashboard?orgId=1" handler=/api/datasources/proxy/uid/:uid/*

(new-dashboard is the name of the dashboard where I try to request Elastic.)

If .JsonData.elasticSearchPath has the good value (which I write in the Configuration page of the plugin), how can I deeply debug my issue ?

Thank you

10

Try enabling the logging option in your [dataproxy] section in your grafana conf file Configure Grafana | Grafana documentation

you also want to put grafana in debug mode which you can do by putting level = debug under the section [log] see Configure Grafana | Grafana documentation

if perhaps you have a github repo with your plugin’s code and tell me what kind of elastic instance I can test it with I can try to reproduce the error and help you fix it.

11

Hello,
Can I change the grafana conf file when I am using a grafana image in my docker-compose file ? If yes, how can I do it ?

Docker compose :

version: '3.0'

services:
  grafana:
    container_name: 'amf-elasticiotoffice-datasource'
    platform: "linux/amd64"
    build:
      context: ./.config
      args:
        grafana_image: ${GRAFANA_IMAGE:-grafana-enterprise}
        grafana_version: ${GRAFANA_VERSION:-10.0.3}
    ports:
      - 3000:3000/tcp
    volumes:
      - ./dist:/var/lib/grafana/plugins/amf-elasticiotoffice-datasource
      - ./provisioning:/etc/grafana/provisioning
    environment:
      - GF_LOG_LEVEL=debug

I can’t send you a github repo because it is a project from my company :frowning:

Thank you

12

Hi @maximedousselaere

From the Grafana docs about configuration you can read it tries to find the file in this location /etc/grafana/grafana.ini

All you have to do is mount that file as a volume, not much different of how you are mounting your dist or provisioning folders.

# ...
  volumes:
      - ./dist:/var/lib/grafana/plugins/amf-elasticiotoffice-datasource
      - ./provisioning:/etc/grafana/provisioning
      - ./grafana.ini:/etc/grafana/grafana.ini
# ...

This will use the file grafana.ini at the same level that your docker compose file.

There’s also a dedicated section to configure grafana with docker in the documentation here Configure a Grafana Docker image | Grafana documentation

13

Hello @academo,

Thank you for your anwser.

I’ve added - GF_DATAPROXY_LOGGING=true in my docker compose file (I couldn’t use the volume in the docker compose file)

In my container logs I have now more details :

logger=datasource t=2024-01-31T12:44:03.478517713Z level=debug msg="Applying default URL parsing for this data source type" type=amf-elasticiotoffice-datasource url=
logger=datasource t=2024-01-31T12:44:03.478528674Z level=debug msg="Data source URL doesn't specify protocol, so prepending it with http:// in order to make it unambiguous" type=amf-elasticiotoffice-datasource url=
logger=data-proxy-log t=2024-01-31T12:44:03.525231726Z level=info msg="Proxying incoming request" userid=1 orgid=1 username=admin datasource=amf-elasticiotoffice-datasource uri=/api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic/ method=POST body="{\"query\":{\"bool\":{\"filter\":[{\"range\":{\"utcDateTime\":{\"gte\": <XXXX>,\"lte\":<XXXXX>,\"format\":\"epoch_millis\"}}},{\"query_string\":{\"analyze_wildcard\":false,\"query\":\"deviceId: <XXXXX>\"}}]}},\"docvalue_fields\":[{\"field\":\"utcDateTime\",\"format\":\"epoch_millis\"}],\"sort\":[{\"utcDateTime\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"size\":500}"
logger=secrets.kvstore t=2024-01-31T12:44:03.525994875Z level=debug msg="got secret value" orgId=1 type=datasource namespace=Elasticiotoffice
logger=data-proxy-log t=2024-01-31T12:44:03.526177754Z level=debug msg=Requesting url=https://<my_company_website>/elastic/
logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic/ remote_addr=172.20.0.1 referer="http://localhost:3000/d/f988b064-36b7-440f-ab53-fa9942d20c67/test?orgId=1" t=2024-01-31T12:44:03.592228345Z level=error msg="Proxy request failed" err="Bad Gateway"
logger=context userId=1 orgId=1 uname=admin t=2024-01-31T12:44:03.592396396Z level=error msg="Request Completed" method=POST path=/api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic/ status=502 remote_addr=172.20.0.1 time_ms=116 duration=116.343422ms size=0 referer="http://localhost:3000/d/f988b064-36b7-440f-ab53-fa9942d20c67/test?orgId=1" handler=/api/datasources/proxy/uid/:uid/*

The log talks about protocol : Data source URL doesn't specify protocol
My {{ .JsonData.elasticUrl }} in the plugin.json file uses HTTPS.
I think I miss a configuration but I don’t know where, could you help me ?

Thank you

14

can you share what you have in your routes? and what is the current final value of jsonData.elasticUrl ?

15

Sure,

In plugin.json :

"routes" : [
    {
      "path" : "elastic",
      "url" : "{{ .JsonData.elasticSearchPath }}",
      "method" : "POST",
      "headers" : [
        {
          "name" : "Accept",
          "content" : "application/json"
        },
        {
          "name" : "Content-Type",
          "content" : "application/json"
        }
      ]
    }
  ]

The call in datasource.ts:

const fullURLbyProxy = this.url + "/elastic/";
   
    const response = await getBackendSrv()
      .fetch({
        url: fullURLbyProxy,
        method: 'POST',
        data: postData,
        headers: this.getESAuthorizationHeaders(),
      })

When I console.log(fullURLbyProxy) I have /api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic/

Value of jsonData.elasticUrl : https://<company-url-for-elasticsearch>/ (I can’t reveal it)

16

@maximedousselaere from the logs and your code it all looks correct. Specially this part logger=data-proxy-log t=2024-01-31T12:44:03.526177754Z level=debug msg=Requesting url=https://<my_company_website>/elastic/ this looks correct.

It seems there might be something wrong in your endpoint and not in the request grafana is doing. Try to debug it on the other end.

Here’s an idea: setup a static server that returns some random json and point your datasource to it. see if the request arrives correctly there.

17

@academo I have tried with JSONPlaceholder - Free Fake REST API (typicode.com) and I have the same error.

When I have the log

2024-02-01T10:24:33.074192282Z logger=data-proxy-log t=2024-02-01T10:24:33.073862682Z level=debug msg=Requesting url=https://<company-url-for-elasticsearch>
2024-02-01T10:24:33.277765421Z logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic remote_addr=172.20.0.1 referer="http://localhost:3000/d/f988b064-36b7-440f-ab53-fa9942d20c67/test?orgId=1" t=2024-02-01T10:24:33.277450317Z level=error msg="Proxy request failed" err="Bad Gateway"
2024-02-01T10:24:33.277845895Z logger=context userId=1 orgId=1 uname=admin t=2024-02-01T10:24:33.277580536Z level=error msg="Request Completed" method=POST path=/api/datasources/proxy/uid/c64d34d0-f36d-49a7-9822-4674b7cb7820/elastic status=502 remote_addr=172.20.0.1 time_ms=208 duration=208.565623ms size=0 referer="http://localhost:3000/d/f988b064-36b7-440f-ab53-fa9942d20c67/test?orgId=1" handler=/api/datasources/proxy/uid/:uid/*

And I go to https://<company-url-for-elasticsearch> (first line) I got my data in my browser.

When I put this URL in my fetch method in datasource.ts file, I got CORS issues. When I put the proxy URL I got 502 issue. I don’t know how to unblock the situation.

18

Hi @maximedousselaere sadly without access to your plugin’s code and way to test the problem I can’t help you debug it further.

What I can recommend you to try is to use one of our plugin examples grafana-plugin-examples/examples/datasource-http at main · grafana/grafana-plugin-examples · GitHub to test the dataproxy and observe how it works.

Also I forgot to ask, are you 100% sure the resources you are trying to access are accessible to your Grafana instance? that it can actually reach those urls?

For example, if you are running grafana in a docker container and you are trying to access http://localhost/elastic-instance while the elastic instance runs in your local machine, it won’t work since grafana’s container localhost is not the same as your host machine. Or if the container doesn’t have dns set up it won’t be able to resolve domains and reach out.

How are you running your grafana instance?

19

Hi* @academo ,

Thank you, I will continue to read and search to work with the proxy.

The ressource is accessible yes.
Actually, my grafana and plugin are running in local.
Elastic is not running in local.
The plugin worked before we migrated elastic from 7.17.2 to v8.9.1.
If elastic is in 8.9.1 we have CORS issues, else we don’t have any issue.

From my local grafana plugin :

  • When I try access the Development environmnent elastic API (v8.9.1) → it fails
  • When I try access the Qualification enrivonment elastic API (v7.17.2) → it works

I’ve tried to access the Qualification elastic with proxy and I have the same 502 issue.
So this means that the problem I have is not from the elastic URL.

My Grafana is running by the docker compose file :

version: '3.0'

services:
  grafana:
    container_name: 'amf-elasticiotoffice-datasource'
    platform: "linux/amd64"
    build:
      context: ./.config
      args:
        grafana_image: ${GRAFANA_IMAGE:-grafana-enterprise}
        grafana_version: ${GRAFANA_VERSION:-10.0.3}
    ports:
      - 3000:3000/tcp
    volumes:
      - ./dist:/var/lib/grafana/plugins/amf-elasticiotoffice-datasource
      - ./provisioning:/etc/grafana/provisioning
    environment:
      - GF_LOG_LEVEL=debug
      - GF_DATAPROXY_LOGGING=true

This is because of the CORS issue I have since the Elastic upgrade that I wanted to send HTTP request to Elastic from the backend part of the plugin, then from the proxy of Grafana (that you advised me).

Thank you.

20

@maximedousselaere try to open an interactive console from the grafana docker container (while grafana runs) docker exec -it amf-elasticiotoffice-datasource bash

and from there try to use curl to fetch the same kind of resource you would do from the grafana instance. See if it works. Also give a try to the jsonplaceholder api curl https://jsonplaceholder.typicode.com/todos make sure the container can access them