How to Monitor Multiple URLs using Prometheus/Grafana

sureshkumarshanmugam · March 27, 2023, 12:27pm

As we are new to this tool, Please help in this regard.

What Grafana version and what operating system are you using? RHEL 8 , Grafana 9.3.6 and Prometheus 2.42.0,Blacbox Exporter 0.43
What are you trying to achieve? - We have bunch of URLs/websites to be monitored in Table format in Grafana
How are you trying to achieve it? - We have configured Prometheus/Blackbox Exporter/Grafana
What happened? We have created Separate Folders for Each team and trying to list only the URLs specific to that team alone by setting the preferences, but the URL’s in prometheus.yml file is taken globally and showing up in other dashboard/Folders too.
What did you expect to happen? - We need to statically show the List of URL’s specific to each and every team that which are configured in the concerned folder
Can you copy/paste the configuration(s) that you are having problems with? We have used the file_sd_configs and using separate files for each team’s URLs
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were. - not seeing Error.
Did you follow any online instructions? If so, what is the URL? - No

sureshkumarshanmugam · March 28, 2023, 3:56am

Hi Team - Please can someone look into this and help us?

fadjar340 · March 28, 2023, 5:28pm

Please paste the prometheus scrape config of your blackbox job…
After analyzed this config, it will be easier to make proper scrape config.

Regards,
Fadjar340

sureshkumarshanmugam · March 29, 2023, 4:47am

Hi - Thanks for looking into this and below is the content what we have currently in prometheus.yml file

scrape_configs:
- job_name: “monitoring job”
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files:
- /home/targets/a_targets.yml
- /home/targets/b_targets.yml
- /home/targets/c_targets.yml
- /home/targets/d_targets.yml
relabel_configs:
- source_labels: [address]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: address
replacement: hostname:9115 #BlackBox Exporter Instance

fadjar340 · April 8, 2023, 8:42am

You can differentiate the job per target and build separate dashboard to each team.

Below the sample for 2 teams:

scrape_configs:
- job_name: “monitoring_job_team_a”
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files:
- /home/targets/a_targets.yml
relabel_configs:
- source_labels: [**address** ]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: **address**
replacement: hostname:9115

- job_name: “monitoring_job_team_b”
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files:
- /home/targets/b_targets.yml
relabel_configs:
- source_labels: [**address** ]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: **address**
replacement: hostname:9115

Then build team a dashboard with filter job=“monitoring_job_team_a”
So with other team, then each team have their own target to watch.

Regards,
Fadjar

sureshkumarshanmugam · April 10, 2023, 3:57am

Thank You, let me try this and post the outcome

sureshkumarshanmugam · April 10, 2023, 4:45am

We are unable to start the Prometheus with the given configuration in Place and below is the error we are facing

ts=2023-04-10T04:36:27.694Z caller=main.go:468 level=error msg=“Error loading config (–config.file=prometheus.yml)” file=//prometheus.yml err=“parsing YAML file prometheus.yml: yaml: line 32: did not find expected alphabetic or numeric character”

Tried with the below options:

relabel_configs:
- source_labels: [address]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: address
replacement: hostname:9115 #BlackBox Exporter Instance

relabel_configs:
- source_labels: [address ] ----> Space after two stars
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: address
replacement: hostname:9115 #BlackBox Exporter Instance

relabel_configs:
- source_labels: [ address ] ---->Spaces begining and after
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: address
replacement: hostname:9115 #BlackBox Exporter Instance

sureshkumarshanmugam · April 10, 2023, 4:46am

Note that the above snippet is missing the ** in the address

fadjar340 · April 10, 2023, 11:02am

Is there any error that prometheus produced?

sureshkumarshanmugam · April 10, 2023, 11:20am

Yes, here it is and it didn’t come up at all

ts=2023-04-10T04:36:27.694Z caller=main.go:468 level=error msg=“Error loading config (–config.file=prometheus.yml)” file=//prometheus.yml err=“parsing YAML file prometheus.yml: yaml: line 32: did not find expected alphabetic or numeric character”

here the line 32 is the [** address ]

fadjar340 · April 10, 2023, 12:02pm

Find the line 32 and check the conformity in the Prometheus documentation. This is yaml file and it’s easy enough to modify. Or you can check using yaml online checker…
Modify as per your environment.
My sample just to show how to differentiate the job to make your dashboard easier to separate.

Regards,
Fadjar

sureshkumarshanmugam · April 10, 2023, 12:15pm

ok thank you for the quick response, will update and let you know the outcome

sureshkumarshanmugam · April 14, 2023, 6:47am

Tried like the above option, but getting an error and the target status is UKNOWN

Error : Invalid : not a valid start token

fadjar340 · April 14, 2023, 12:54pm

Probably you need to remove all the “*” in the address, I saw that * make bold previous post…

sureshkumarshanmugam · April 18, 2023, 5:00am

Tried that too, seeing the below error

Error : Invalid : not a valid start token

fadjar340 · April 18, 2023, 7:47am

Check the log using tail -f then try to restart prometheus.
Please paste the log here…

sureshkumarshanmugam · April 20, 2023, 9:00am

Hi @fadjar340 - Thanks for your Help, not seeing the Error