Loki errors when viewing some alert rules history

[ewane420]

Thank you Kimberly. I’m on my way to Owatonna Minnesota now to discuss this further.

[kimberlyyerhot63]

Okay sounds great

[scootertramp]

Update: Reverted loki_max_query_length back to 721h..

@scootertramp

loki_max_query_length to 90h from 721h.

Basically this issue was a result of trying to solve a Grafana issue with
https://community.grafana.com/t/error-saving-alert-annotation-batch/162761/11

I followed a recommendation to send alert state history to Loki.
This is a small home system with a monolithic (single-binary) Grafana Loki deployment.
I had been running well using Loki as a syslog server that maintained 60 days of logs at
about 500MB of data.

Currently I am trying to reduce the amount of state history.
I have started with the following and with adjust to see how much works best.

   retention_stream:
    - selector: '{from="state-history",group="Prometheus"}'
      priority: 1
      period: 72h

    - selector: '{from="state-history",group="SSH-Connections"}'
      priority: 1
      period: 72h

    - selector: '{from="state-history",group="Rsyslog"}'
      priority: 1
      period: 72h

[infofcc3]

Thanks for the update. Since reverting loki_max_query_length to 721h didn’t immediately bring the errors back, it does seem that loki_max_query_length itself wasn’t the root cause.

Limiting retention for the from="state-history" streams is a reasonable next step. If reducing the retained alert state history consistently prevents the scheduler_processor.go context canceled/EOF errors, that would suggest the issue is related to the amount of state-history data being queried.

It would be interesting to see whether the errors remain gone after you’ve run with the shorter retention for a few days.

[scootertramp]

Just upgraded Loki to latest version.

pi@raspberry05:~ $ loki -version
loki, version 3.7.4 (branch: release-3.7.x, revision: b318f282)
  build user:       root@9dc424156f20
  build date:       2026-07-22T03:46:43Z
  go version:       go1.26.5
  platform:         linux/arm64
  tags:             netgo

Currently waiting for new retention levels to settle.

 retention_period: 168h

 retention_stream:
    - selector: '{from="state-history",group="Prometheus"}'
      priority: 1
      period: 72h                  # Prometheus Alerts are wiped after 3 Days

    - selector: '{from="state-history",group="SSH-Connections"}'
      priority: 1
      period: 72h                  # SSH Connection Alerts are wiped after 3 Days

    - selector: '{from="state-history",group="Rsyslog"}'
      priority: 1
      period: 72h                  # Rsyslog Alerts are wiped after 3 Days

Current Loki storage is:

pi@raspberry05:~ $ sudo du -h -d 1 /opt/loki
456K    /opt/loki/retention
66M     /opt/loki/chunks
1.5M    /opt/loki/tsdb-shipper-active
632K    /opt/loki/wal
968K    /opt/loki/tsdb-shipper-cache
4.0K    /opt/loki/rules
70M     /opt/loki

Down from 500MB.

[scootertramp]

So far since stabilizing at my current retention levels, I have not seen the Loki errors.

retention_period: 168h and  retention_streams set to 72H

caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=127.0.0.1:9096

Current Loki storage usage:

pi@raspberry05:~ $ sudo du -h -d 1 /opt/loki
1.5M    /opt/loki/retention
77M     /opt/loki/chunks
1.5M    /opt/loki/tsdb-shipper-active
536K    /opt/loki/wal
1.1M    /opt/loki/tsdb-shipper-cache
4.0K    /opt/loki/rules
82M     /opt/loki

This is my current Alert State History levels.

I am going to attempt to set retention levels as follows.

 retention_period: 720h

 retention_stream:
    - selector: '{from="state-history",group="Prometheus"}'
      priority: 1
      period: 168h                  # Prometheus Alerts are wiped after 7 Days

    - selector: '{from="state-history",group="SSH-Connections"}'
      priority: 1
      period: 168h                  # SSH Connection Alerts are wiped after 7 Days

    - selector: '{from="state-history",group="Rsyslog"}'
      priority: 1
      period: 168h                  # Rsyslog Alerts are wiped after 7 Days
```

So I am inclined to believe how Grafana queries Loki when you look at the history page
of an alert is what is causing the error. I don’t see the error when looking at the main history
page as it defaults to just 1 hour for a time period.

[kimberlyyerhot63]

Thank you for figuring that out. I hope that things continue to run at what it is supposed to..

[scootertramp]

A few hours later.

When looking at alert history for Rsyslog alerts.

Only a small increase in records for history of this alert.

[scootertramp]

Updated to latest version.

Upgraded Loki:

pi@raspberry05:~ $ loki -version
loki, version 3.7.6 (branch: release-3.7.x, revision: 5003600d)
  build user:       root@31d37f362fe2
  build date:       2026-08-05T15:36:54Z
  go version:       go1.26.5
  platform:         linux/arm64
  tags:             netgo

Current Loki storage:

pi@raspberry05:~ $ sudo du -h -d 1 /opt/loki
456K    /opt/loki/retention
118M    /opt/loki/chunks
1.5M    /opt/loki/tsdb-shipper-active
500K    /opt/loki/wal
1.2M    /opt/loki/tsdb-shipper-cache
4.0K    /opt/loki/rules
122M    /opt/loki

Still waiting on new streams retention to fill out. (168h)

Current Alert State History levels: