Hi, we are running Loki in microservices mode in the Kubernetes cluster and getting a problem with the Ruler component. The alerting rules work incorrectly when the time range of the alert expression is more than probably ~30 minutes. e.g sum(count_over_time({service="my-service"} |= "sign-in" [6h])) < 10
In this case I get a fake alert because Ruler can’t to get the results of the query for the entire time window 6h. As a persistent storage backend, we use Amazon S3 for storing chunks. As I’m understanding it correctly the Ruler can’t reach the Amazon S3 bucket and just gets the query results from the ingesters. Does this problem happen with someone? Is this a known issue?