Loki flush issue using loki-simple-scalable with single gcs

Grafana Loki
1

Hi Loki Team,

Using Loki 2.5 - loki-simple-scalable via helm chart
loki-simple-scalable-1.0.0 2.5.0

We encountered the high load issue and loki-write will be crash becuause of file system full .
The error in loki-write pod was

level=error ts=2022-11-28T03:45:21.648048478Z caller=flush.go:222 org_id=log-pr msg=“failed to flush user” err=“googleapi: Error 429: The rate of change requests to the object loki-store-002/log-pr/e351c4231844d077:1843ad0cf4e:1843ad3b52c:1e4fdb5d exceeds the rate limit. Please reduce the rate of create, update, and delete requests., rateLimitExceeded”
level=error ts=2022-11-28T03:45:21.953071575Z caller=flush.go:222 org_id=log-pr-2 msg=“failed to flush user” err=“googleapi: Error 403: gcs-sa@log-pr-0b9060.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object., forbidden”
level=error ts=2022-11-28T03:45:22.027523705Z caller=flush.go:222 org_id=log-pr msg=“failed to flush user” err="googleapi: Error 403:

When the load is low, it’s working fine.

Thanks,
Ed

2

Do we need to create a new loki clsuter for different tenant to use different gcs bucket?
Error - "The rate of change requests to the object loki-store-002/log-pr/e351c4231844d077:1843ad0cf4e:1843ad3b52c:1e4fdb5d exceeds the rate limit. Please reduce the rate of create, update, and delete requests., rateLimitExceeded”

Thanks,
Ed

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.