Alert Issue in Dashboard with Prometheus Dashboard: "request handler error: Post "http://localhost:9090/api/v1/query_range": dial tcp [::1]:9090: connect: connection refused"

Grafana Cloud
1

Hello, I am trying to create alerts on a Grafana dashboard using some analytics from Prometheus. I have successfully been able to connect my Prometheus data source to Grafana and am able to view the data. I know that this is not an issue with how I am configuring alert rules in general or the server itself because I am receiving alerts related to other metrics in the same dashboard, and all of the data for every metric is showing up. I am specifically running into an issue with creating an alert related to the ‘up’ metric of Prometheus, which lets me know if my server is down.

  • What Grafana version and what operating system are you using?
    Grafana Cloud, Windows

  • What are you trying to achieve?
    Trying to create an alert related to the ‘up’ metric of Prometheus in a Grafana dashboard. I am running Prometheus on my local machine and am using it to grab analytics from a website.

  • How are you trying to achieve it?

    image
    image1451×900 101 KB

image
image1156×692 29.5 KB

Server address is blacked out in last images.

  • What happened?
    Received this error after using ‘test rule’ to test the alert rule for the up metric.
{
  "firing": true,
  "state": "pending",
  "conditionEvals": " = true",
  "timeMs": "5.406ms",
  "error": "request handler error: Post \"http://localhost:9090/api/v1/query_range\": dial tcp [::1]:9090: connect: connection refused",
  "logs": [
    {
      "message": "Condition[0]: Query",
      "data": {
        "from": blocked out,
        "queries": [
          {
            "refId": "A",
            "model": {
              "exemplar": true,
              "expr": "up{instance=\"blocked out\", job=\"server\"}",
              "interval": "",
              "legendFormat": "",
              "queryType": "randomWalk",
              "refId": "server"
            },
            "datasource": {
              "id": 18,
              "name": "Prometheus"
            },
            "maxDataPoints": 0,
            "intervalMs": 0
          }
        ],
        "to": blocked out
      }
    }
  ]
}

  • What did you expect to happen?
    I did not expect to receive an error and expected to be alerted that the server was down (as indicated by the ‘0’) in the graphic.

  • Can you copy/paste the configuration(s) that you are having problems with?
    Shown above.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Shown above.

1 Like
2

Hi @deeptanshadwivedi –

Taking a look at your screenshots (without knowledge on your account info) it appears that although you note that you’re using Grafana Cloud you’re not using the Grafana Cloud hosted Prometheus endpoint, rather you’ve just added an external Prometheus data source which is connected via HTTP to your external Prometheus location. Are you aware that we offer Prometheus metrics hosting (even in a forever Free tier)? Thought I’d point that out first.

That said, if you wish to continue hosting that Prometheus data externally, then the issue you’re experiencing is not specific to Grafana Cloud since its not actually interacting with our Prometheus services, just the Grafana interface on the Cloud which functions largely the same as an OSS Grafana, particularly when it comes to simply connecting to external data sources.

The error you’re receiving looks based in the HTTP call to your external Prometheus (localport:9090). It would point at either a network error or some error in the path. I’ve found this issue where the user reported a similar error and resolved it: Issue. Please review the comments (some from our internal Grafanistas) and see if those are helpful to you in troubleshooting this error. That particular user resolved the issue by replacing the URL in the Grafana UI Prometheus data source from http://localhost:9090 to http://server-ip:9090.

5 Likes
3

Hi @ximenaaliaguilla I ran into the same issue as @deeptanshadwivedi and your solution fixed my issue. I was using Grafana Cloud and not using the Grafana Cloud Hosted Prometheus endpoint located at http://localhost:3000

I also found that for first time login to http://localhost:3000 both the username and password is “admin” (take out “”)

Thank you for your help! @ximenaaliaguilla