I am testing Alloy to scrape metrics from a locally installed and running script_exporter:
prometheus.scrape "script_my_script" {
targets = [ {"__address__" = "localhost:9469"} ]
scrape_interval = "1m"
scrape_protocols = ["PrometheusProto", "OpenMetricsText1.0.0", "OpenMetricsText0.0.1", "PrometheusText0.0.4"]
extra_metrics = true
metrics_path = "/probe"
params = { "script" = ["my_script"] }
forward_to = [prometheus.remote_write.default.receiver]
}
This works so far, I can see in the script_exporter logs the successful scraping in the defined interval, coming from Alloy.
The output from script_exporter several shows metrics in Prometheus format:
# HELP script_success Script exit status (0 = error, 1 = success).
# TYPE script_success gauge
script_success{script="my_script"} 1
# HELP script_duration_seconds Script execution time, in seconds.
# TYPE script_duration_seconds gauge
script_duration_seconds{script="my_script"} 6.051769
# HELP script_exit_code The exit code of the script.
# TYPE script_exit_code gauge
script_exit_code{script="my_script"} 0
# HELP script_use_cache Script use cache (0 = no, 1 = yes).
# TYPE script_use_cache gauge
script_use_cache{script="my_script"} 0
#HELP my_script_hardware_health Overall health status of server. 0=OK, 1=Warning, 2=Critical
#TYPE my_script_hardware_health gauge
my_script_hardware_health 0
#HELP my_script_hardware_health_temperatures Health status of hardware component Temperatures
#TYPE my_script_hardware_health_temperatures gauge
my_script_hardware_health_temperatures 0
However where are the scraped metrics of the script’s output? I can’t find them in the metrics store.
The documentation says:
The scraped metrics are forwarded to the list of receivers passed in
forward_to
.
So I’m expecting to see these metrics, as seen and scraped, in the metrics store and viewable in Grafana’s Explore.
Under debug metrics I came across this:
prometheus_forwarded_samples_total
(counter): Total number of samples sent to downstream components.
I can display the following metric in Grafana Explore and if I interpret this metric correctly, it seems that there were metrics (samples?) collected from the script_exporter output and forwarded.
prometheus_forwarded_samples_total{component_id="prometheus.scrape.script_my_script",component_path="/",instance="0.0.0.0:12345",job="alloy"} 260
However I also came across another metric scrape_samples_post_metric_relabeling
which shows 0 samples:
scrape_samples_post_metric_relabeling{instance="localhost:9469",job="prometheus.scrape.script_my_script"} 0
Could it have something to do that scrape_samples_post_metric_relabeling
shows 0 samples (metrics?) after relabeling? Are the scraped metrics lost due to a relabeling?
So TL;DR: Scraping works, but scraped metrics (from the script_exporter output) are not stored. Or are they and I’m looking in the wrong place/metric? Couldn’t figure this out so far…
Any help is appreciated, thanks