(First) Prometheus data source fails

Grafana Prometheus
1

  • What Grafana version and what operating system are you using?

    Grafana 9.3.2 / Ubuntu 20.04 (TIG stack)

  • What are you trying to achieve?

    Trying to add a Prometheus data source

  • How are you trying to achieve it?

  • What happened?

    • UI error: Error reading Prometheus: An error occurred within the plugin
    • Journal log:
Feb 08 23:17:14 tig.domain.com grafana-server[1062]: logger=context userId=3 orgId=1 uname=user t=2023-02-08T23:17:14.815101318+01:00 level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: received empty response from prometheus" remote_addr=127.0.0.1 traceID=
Feb 08 23:17:14 tig.domain.com grafana-server[1062]: logger=context userId=3 orgId=1 uname=user t=2023-02-08T23:17:14.815174998+01:00 level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=2 duration=2.286542ms size=116 referer=https://grafana.domain.com/datasources/edit/xhAng9A4k handler=/api/ds/query
Feb 08 23:17:22 tig.domain.com grafana-server[1062]: logger=cleanup t=2023-02-08T23:17:22.422436072+01:00 level=info msg="Completed cleanup jobs" duration=3.860605ms

  • What did you expect to happen?

    I expected to see a connection on the remote server hosting the Prometheus URL. Instead, I see Grafana trying to connect to localhost and failing. When I use curl from the Grafana server I can connect so it’s not a firewall or connectivity issue.

  • Can you copy/paste the configuration(s) that you are having problems with?

    n/a

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

    See above

  • Did you follow any online instructions? If so, what is the URL?

    I’m trying to get this data source working in Grafana: GitHub - camptocamp/ipahealthcheck_exporter: Prometheus exporter for exposing ipa-healthcheck tool checks as metrics

2

You are connecting to Prometheus Exporter. But Grafana needs to connect to Prometheus Server.

Logic:
1.) Prometheus Exporter expose some metrics (usually on /metrics)
2.) Prometheus Server collects metrics from Prometheus Exporter and store them
3,) Grafana connects to Prometheus Server and read/visualize stored metrics.

2 Likes
3

Ah, now things make perfect sense. Thank you for the succinct explanation!

Rather than continue my attempts I’ve opted for ingesting the exporter metrics using Telegraf into InfluxDB. Reading up on Prometheus Server, my understanding now is that it would essentially add another telemetry storage location when I already have one.

4

Hi @jangaraj thanks for this.

I’m having a similar problem and suspect your solution may tap into it.
For context: I exposed my Prometheus server to port 9090. going to http://localhost:9090 lands me on the Prometheus Time Series Collection and Processing Server web page and going to http://localhost:9090/metrics prints a bunch of metrics on the browser. However, when I go to my community.grafana.com (where my grafana is exposed) and try to add a Prometheus DataSource with URL http://localhost:9090 it blinks green, then red, and gives me this error: Error reading Prometheus: An error occurred within the plugin

Per your explanation above, http://localhost:9090 should be the Prometheus server instead of http://localhost:9090/metric is that right?

I highly appreciate any insights.

If it helps, here is the output of curl:
`curl http://localhost:9090
Output:

Found.

7

I have got the issue fixed with Custom HTTP Headers authentication approach that header value is Authorization and value is Bearer

8

Hi @edsonaguiar17

I faced the same problem and solved changing http://localhost:9090 to http://prometheus:9090.
Hope it works for you too!

1 Like