Update: Reverted loki_max_query_length back to 721h..
Basically this issue was a result of trying to solve a Grafana issue with
https://community.grafana.com/t/error-saving-alert-annotation-batch/162761/11
I followed a recommendation to send alert state history to Loki.
This is a small home system with a monolithic (single-binary) Grafana Loki deployment.
I had been running well using Loki as a syslog server that maintained 60 days of logs at
about 500MB of data.
Currently I am trying to reduce the amount of state history.
I have started with the following and with adjust to see how much works best.
retention_stream:
- selector: '{from="state-history",group="Prometheus"}'
priority: 1
period: 72h
- selector: '{from="state-history",group="SSH-Connections"}'
priority: 1
period: 72h
- selector: '{from="state-history",group="Rsyslog"}'
priority: 1
period: 72h