Prometheus.scrape working, but where are the scraped metrics?

I am testing Alloy to scrape metrics from a locally installed and running script_exporter:

prometheus.scrape "script_my_script" {
  targets = [ {"__address__" = "localhost:9469"} ]
  scrape_interval = "1m"
  scrape_protocols = ["PrometheusProto", "OpenMetricsText1.0.0", "OpenMetricsText0.0.1", "PrometheusText0.0.4"]
  extra_metrics = true
  metrics_path = "/probe"
  params = { "script" = ["my_script"] }
  forward_to = [prometheus.remote_write.default.receiver]
}

This works so far, I can see in the script_exporter logs the successful scraping in the defined interval, coming from Alloy.

The output from script_exporter several shows metrics in Prometheus format:

# HELP script_success Script exit status (0 = error, 1 = success).
# TYPE script_success gauge
script_success{script="my_script"} 1
# HELP script_duration_seconds Script execution time, in seconds.
# TYPE script_duration_seconds gauge
script_duration_seconds{script="my_script"} 6.051769
# HELP script_exit_code The exit code of the script.
# TYPE script_exit_code gauge
script_exit_code{script="my_script"} 0
# HELP script_use_cache Script use cache (0 = no, 1 = yes).
# TYPE script_use_cache gauge
script_use_cache{script="my_script"} 0
#HELP my_script_hardware_health Overall health status of server. 0=OK, 1=Warning, 2=Critical
#TYPE my_script_hardware_health gauge
my_script_hardware_health 0
#HELP my_script_hardware_health_temperatures Health status of hardware component Temperatures
#TYPE my_script_hardware_health_temperatures gauge
my_script_hardware_health_temperatures 0

However where are the scraped metrics of the script’s output? I can’t find them in the metrics store.

The documentation says:

The scraped metrics are forwarded to the list of receivers passed in forward_to.

So I’m expecting to see these metrics, as seen and scraped, in the metrics store and viewable in Grafana’s Explore.

Under debug metrics I came across this:

  • prometheus_forwarded_samples_total (counter): Total number of samples sent to downstream components.

I can display the following metric in Grafana Explore and if I interpret this metric correctly, it seems that there were metrics (samples?) collected from the script_exporter output and forwarded.

prometheus_forwarded_samples_total{component_id="prometheus.scrape.script_my_script",component_path="/",instance="0.0.0.0:12345",job="alloy"} 260

However I also came across another metric scrape_samples_post_metric_relabeling which shows 0 samples:

scrape_samples_post_metric_relabeling{instance="localhost:9469",job="prometheus.scrape.script_my_script"} 0

Could it have something to do that scrape_samples_post_metric_relabeling shows 0 samples (metrics?) after relabeling? Are the scraped metrics lost due to a relabeling?

So TL;DR: Scraping works, but scraped metrics (from the script_exporter output) are not stored. Or are they and I’m looking in the wrong place/metric? Couldn’t figure this out so far…

Any help is appreciated, thanks :slight_smile:

Typical. I collect data, do nice formatting, obfuscate sensitive data and create the topic here in the forums - and then the metrics show up in the Grafana Explorer.

So it works. I was probably just a bit too impatient. :man_shrugging:

The scrape_samples_post_metric_relabeling metric has meanwhile also a value by the way:

scrape_samples_post_metric_relabeling{instance="localhost:9469",job="prometheus.scrape.script_my_script"} 15
1 Like