Configure grafana agent to send cpu metric every second?

Grafana Grafana Agent
,
1

I have the below configuration. I would like to scrape the cpu metric every second and then I have Grafana set up to auto refresh every 5s. The problem is that there seems to be a lag between when I set the CPU to a value (i.e. 100%) and when the metrics show up in Prometheus. The lag is about 30 seconds. Once the metrics DO show up in Prometheus, Grafana will update previous values displayed int he graph to the correct values (so basically 30 seconds after the time period in the graph the data is correct but before that it’s showing lagged data). How Can I update grafana agent to send the metric faster?

prometheus.exporter.windows "immediate" { 
    enabled_collectors = ["cpu"]
}

prometheus.scrape "windows_immediate_scraper" {
  scrape_interval = "1s"
  scrape_timeout = "1s"
  targets    = prometheus.exporter.windows.immediate.targets
  forward_to = [prometheus.relabel.set_instance_to_hostname.receiver]
}

////////////// RELABELER (to set "Instance" to machine name) //////////////////////
prometheus.relabel "set_instance_to_hostname" {
  forward_to = [prometheus.remote_write.default.receiver]
  
  rule {
    action        = "replace"
    source_labels = ["instance"]
    replacement = constants.hostname
	target_label = "instance"
  }
}

////////////////////////////// Prometheus writer /////////////////////////////////////////
prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/v1/write" //Prometheus
  }
}

and then I am executing the promql query:

100 - (avg by (instance) (irate(windows_cpu_time_total{instance="MSSQL-MGMT",mode="idle"}[1s])) * 100)

2

bump anybody?

3

Preface by, 1 scrape per second is pretty high and not recommended.

Are you sure the Agent is not scraping once per second? You can generally look at the logs in debug mode and see a scrape occurring. Accessing the UI will also let you see last scrape time and you can refresh to ensure its scraping roughly every second. Grafana dashboards tends to do some interpolation/assumptions on scrape interval so its possible your seeing this behavior.

Metrics are also queued and may not be sent to the remote write endpoint in 1 second intervals.

4

Thank you for replying!

ok yes, I should actually rephrase (or rather, temper expectations). The scrape is happening every 1 second, but how to do I get the remote_write to occur…more frequently? Grafana can at minimum do 5 second refresh, how do I get it to send that fast? I can see in the Grafana agent dashboard that it IS scraping very frequently (< 5s at the least), but yet when I query prometheus it takes 30 seconds or more for the data to show up. I do not know how to see how often the remote_write is sending to prometheus or if it’s that the scraper isn’t sending it to the remote writer often enough…?

5

batch_send_deadline in prometheus.remote_write | Grafana Agent documentation.

6

ok, yes I saw that and will go ahead and set that and see. I see it defaults though to 5s so I was unsure if that was an issue since, as I said before, it can take 30+ seconds for the data to arrive. I’ll let you know when I test. Thanks!

7

ok, I tested this - it’s still about a 30s delay between when I set the CPU to 100% and when that data shows up in Prometheus/Grafana dashboard