Grafana is showing Website is down while my website is up

[adityaproeffico]

I’m running Grafana version 9.2.3 on Ubuntu 20, and I’m trying to monitor a website using the Prometheus Blackbox exporter and the probe_http_status_code module. The website seems to be working fine, and Prometheus is reporting that it’s up, but Grafana is showing that it’s down. I’ve uploaded my Blackbox configuration and Prometheus configuration.

I’m new to Grafana and could use some help troubleshooting this issue. Can you please provide guidance on how to identify and fix the problem? Thank you!

Website status:

Grafana:

1. blackbox yml:

modules:
  http_2xx:
    prober: http
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  http_post_2xx:
    prober: http
    http:
      ip_protocol_fallback: true
      method: POST
    tcp:
      ip_protocol_fallback: true
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  icmp:
    prober: icmp
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  irc_banner:
    prober: tcp
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
      query_response:
      - send: NICK prober
      - send: USER prober prober prober :prober
      - expect: PING :([^ ]+)
        send: PONG ${1}
      - expect: ^:[^ ]+ 001
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  pop3s_banner:
    prober: tcp
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
      query_response:
      - expect: ^+OK
      tls: true
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  ssh_banner:
    prober: tcp
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
      query_response:
      - expect: ^SSH-2.0-
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true
  tcp_connect:
    prober: tcp
    http:
      ip_protocol_fallback: true
    tcp:
      ip_protocol_fallback: true
    icmp:
      ip_protocol_fallback: true
    dns:
      ip_protocol_fallback: true

2. prometheus yml

- job_name: blackbox
  honor_timestamps: true
  params:
    module:
    - http_2xx
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  follow_redirects: true
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: localhost:9115
    action: replace
  static_configs:
  - targets:
    - https://prolificapp.in
    - https://shreemarutinandan.com
    - https://procliniq.in
    - https://pro.ogaanlam.com
    - https://prolificapp.in
    - https://shreemarutinandan.com
    - https://procliniq.in
    - https://pro.ogaanlam.com
    - https://fundbridge.in
    - https://admin.fundbridge.in
    - https://ops360.live
    - https://live.shreemarutinandan.com
    - https://fueasia.org
    - https://martnmore.in
    - https://walkinmart.in
    - https://api.prolificapp.in
    - https://api.fundbridge.in
    - https://api.procliniq.in
    - https://api.fueasia.org
    - https://needs24.prolificapp.in
    - https://api.ogaanlam.com
    - https://77.martnmore.in/mm77
    - https://93.martnmore.in
    - https://120.martnmore.in
    - https://122.martnmore.in
    - https://live.urbngrocers.com
    - https://77.martnmore.in/mm77
    - https://122.martnmore.in
    - http://needs93.prolificapp.in
    - https://mail.proeffico.email
    - https://vhstechnology.in
    - https://thecuriousaders.com
    - https://swachhbharatcaptains.in
    - https://deepikafoi.in

[pranavgaur]

Hello @adityaproeffico; the same thing happened with my setup. I am running my Grafana using a docker container and monitoring the metrics of other containers and websites. At a particular time today (at IST 6:00) all of my websites url’s started giving me status code 0 and my inbox flooded with down alerts even though the websites had status 200. Once I restarted the Grafana and black-box container it still didn’t work but after some time the issue automatically got resolved.

The cause of this is still unknown but looking forward to getting any other help regarding this.

[jangaraj]

The best way to find root cause is run the same test on blackbox manually with debug=true parameter. Blackbox probe will return logs and all generated metric, which give you a clue, e. g. timeout, dns resolution problem,…

It is not a grafana problem. Grafana just reports on data, which are in prometheus DB.

Example ( URL and parameters must reflect your blackbox setup):

[fadjar340]

In the first image you use GET method, but your blackbox yaml use POST.
I suggest you change the yaml like below:

modules: 
  http_2xx: 
    http: 
      fail_if_not_ssl: true
      ip_protocol_fallback: false
      method: GET
      no_follow_redirects: false
      preferred_ip_protocol: ip4
      valid_http_versions: 
        - HTTP/1.1
        - HTTP/2.0
    prober: http
    timeout: 15s

Regards,
Fadjar