Hello, this is my first Community post, so I’ll do my best to be descriptive and succinct.
I’m using Loki 2.8x and Prometheus Alert Manager. I’m using local storage for my alerting rules, storing them in a yaml file in my file system at /etc/{application-name}/rules/namespace. I’ve written some code locally that can add and remove rules from the yaml file, and I want the Alert Manager to detect and evaluate these new rules without having to restart Loki. Sometimes, the Alert Manager will detect the new rule and begin firing without my having to restart Loki. Other times, it won’t, even when I can see that the logs that should get it to fire are present. Through testing, I’ve noticed that the new rules tend to be evaluated when a new rule Group is added, and not when a new rule is added under an existing group.
The rules I’m attempting to set up are very simple: I just want to scan logs for a given keyword within a given timeframe. If I write a new rule to search for more than 0 instances of the word “Error” in the last minute, and I observe that the rule is now in my yaml file, and I see that there are logs within the last minute that contain the word “Error,” I expect the Alert to be firing. To reiterate: sometimes the new rules are evaluated and fire without restarting Loki, other times they are not.
All of the digging through the docs that I’ve done tell me that my issue is that Loki does not hot-reload rules while using local storage, and that I need to restart Loki for the new rules to be evaluated. Hot-reloading is available when using the Ruler API, but the API does not support local storage. I understand that using the API would allow me to call endpoints to add and edit my rules, but I don’t want to have to use an s3 bucket (or similar cloud storage) to store my rules file-- I just want it to be local. The docs state that local storage is read-only, and is not support rule creation or deletion through the Ruler API. Can I create/delete the rules using my own local code, and use the Ruler API to evaluate (read) my local file and check for changes?
In summary, my question to the community is: is there a way for me to hot reload the rules file without restarting Loki while using local storage for the rules file?
Thank you, and please let me know if any additional information about my configuration is needed.