Alloy Samples Sent steadily increasing over time — how to identify the cause and keep it stable?

Grafana Loki
1

Hey folks,

I’m seeing Alloy Samples Sent keep creeping up over time instead of staying flat. It’s not a one-off spike — it’s a steady rise, and it’s been forcing us to keep adjusting alert thresholds, which is getting noisy and hard to manage.

What I’m seeing:

  • Samples Sent is trending upward over days/weeks.

  • The rise is spread across multiple instances, not just one bad pod.

  • scrape_samples_scraped is rising more than scrape_samples_post_metric_relabeling, so it looks like the problem is happening before relabeling.

  • prometheus_remote_storage_samples_pending has spikes, while prometheus_remote_storage_shards_desired stays pretty low.

What I want to understand:

  • What usually causes this kind of steady increase?

  • Is it target growth, duplicate scraping, cardinality growth, or remote-write backlog?

  • What’s the best way to prove the actual root cause?

  • How do people keep this metric stable so alerts don’t need constant threshold changes?

I’ve already checked that it doesn’t look like one isolated scrape job or one endpoint. The namespace-level view also doesn’t feel reliable enough to explain it properly.

If anyone has seen this before, what should I check next? What queries or Alloy metrics would actually help narrow it down?