Can't publish snapshots to a custom external server

Grafana Configuration
1

I have configured a Grafana server to publish snapshots to an external Grafana server by setting the following in grafana.ini:

external_enabled = true
external_snapshot_url = http://www.xyz.com:3000

When I try to publish a snapshot to the external server I get an “Unexpected error” in the UI. After further investigations it seems that the first OPTIONS request to api/snapshots returns a 404 with body {“message”:“Not found”}.

I can successfully publish local snapshots on both Grafana servers. Also, from both servers I can publish a snapshot of the same dashboard to snapshot.raintank.io.

Is there anything I’m missing in my configuration? What could be causing this error?

2

Hi,

I think that you’ll need to enable CORS on your external grafana instance. Easiest way of doing this is probably by running a reverse proxy that’s responds with proper CORS responses.

Marcus

3

Hi
I am having the problem. Why would you think ia reverse proxy will fix the problem?
As it is an issue between two different applications?
I have a company openshift stack so difficult to add extra components.

My debugging is limited, I just see a , no message, in the chrome network tab, so not much to go on.

Thanks
Anthony

4

You’ll need the proxy to properly respond to HTTP OPTIONS requests (CORS preflight requests).

5

Hi Marcus,
Would you have an example of a reverse proxy setting,i am still struggling, going from a localhost grafana to a remote [called remote-grafana]

looking for general Cors solution, just a demo i am doing, so not too worried about security for now.

< Location / >
proxyPass http://remote-grafana/
ProxyPassReverse http://remote-grafana/
RequestHeader set Host “remote-grafana”
ProxyPreserveHost On

	Header always set Access-Control-Allow-Origin "*"

Header add “Access-Control-Allow-Origin” “"
#Header Set Access-Control-Allow-Origin "”
< / Location >

also,just to clarify: The dashboard I want to snapshot only exists on the local grafana not the remote grafana,

and my apache is on my local grafana, not the remote one.

Thanks
anthony

6

Here’s an nginx example config used for one of our graphite docker container when developing grafana: https://github.com/grafana/grafana/blob/master/docker/blocks/graphite1/conf/etc/nginx/sites-enabled/graphite-statsd.conf

If you’re using apache it shouldn’t be that hard to map CORS config from nginx to apache

Marcus

7

No luck yet, guess have to turn on full apache logging tomorrow.
just to check, on the remote grafana which will store the snapshots, it is just the default configuration, i have not changed any settings as i read snapshots is on by default

8

I switched to nginx as my ubuntu apache reverse proxy had a known bug when the url is longer than 98 characters , a url truncation is done (i have a long openshift url)

looking at nginx logs:
http upstream request: “/api/snapshots?”

my local grafana tries to copy the snapshot to this remote grafana, gets a 404 response code, but does it need an API key for the Authorization: Bearer , into the remote grafana? or does the remote grafana need to be put into annoyomous mode or something.

i am blind at the moment, knowing what/where thew problem/blockage is?

thanks

9

I think you need to add authentication (api key or basic authentication) information to an incoming request in nginx before you forward the request to to remote grafana.

If you still have issues you need to provide more detailed information. Like the complete request (including header and payload) from local grafana when saving snapshot and the complete response from (including header and payload) from that request.

You should also be able to check nginx log and the remote grafana server log - you can enable debug logging and router logging.

Marcus

10

Thanks very much Marcus ,it is all working now.
I have a local Grafana, where I generate the snapshots ,and a remote Grafana to receive the snapshots (called remote-grafana.com for documentation), with NGINX running locally .

For the local Grafana I have:
[snapshots]
external_enabled = true
external_snapshot_url = http://localhost
external_snapshot_name = Publish to company

Then for nginx I have:

defaults.ini

server {
listen 80;

index index.html;
location / {
proxy_pass http://remote-grafana.com/;

proxy_set_header Host remote-grafana.com";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header ‘Access-Control-Allow-Origin’ ‘*’;
add_header ‘Access-Control-Allow-Methods’ ‘GET, POST, OPTIONS’;
add_header ‘Access-Control-Allow-Headers’ ‘Authorization, Content-Type’;
add_header ‘Access-Control-Allow-Credentials’ ‘true’;

proxy_set_header Authorization “Bearer eyJrIjoiSlZjeFgxbm5YYUhYUGVGZkxWaW4xaXA2WWhVVkNBdWciLCJuIjoiYWRtaW4yIiwiaWQiOjF9”;

}
}

The reason why NGINX is needed I found was:
The private grafana issues a /api/snapshots OPTIONS api call to the remote Grafana before doing a POST api, I am not sure why, as it is not needed for a remote server, but it happens.
The API KEY for the public facing Grafana is inserted by the nginx changes. for securityI can improve this later and add HTTPS to the Grafana’s.

Anthony

2 Likes