ApiException when calling kubernetes: (403)\nReason: Forbidden

We installed loki in a EKS cluster and we get this error in loki-backend pods:

{“time”: “2023-10-04T13:00:15.390666+00:00”, “msg”: “ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({‘Audit-Id’: ‘6a0d82b2-f55c-4e0f-b363-14629bef4c26’, ‘Cache-Control’: ‘no-cache, private’, ‘Content-Type’: ‘application/json’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Kubernetes-Pf-Flowschema-Uid’: ‘5bbf862c-c5d8-4d18-a701-96094703d771’, ‘X-Kubernetes-Pf-Prioritylevel-Uid’: ‘c9225b65-899a-4646-8c13-ab29a339cd5b’, ‘Date’: ‘Wed, 04 Oct 2023 13:00:15 GMT’, ‘Content-Length’: ‘283’})\nHTTP response body: b’{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets is forbidden: User \\"system:serviceaccount:loki:loki\\" cannot watch resource \\"secrets\\" in API group \\"\\" in the namespace \\"loki\\"","reason":"Forbidden","details":{"kind":"secrets"},"code":403}\n’\n\n”, “level”: “ERROR”}

Hi,
I think this is the sc-rules sidecar of the backend pods which reports these errors.
It tries to get secrets/configmap in all the namespaces (by default) and the loki ServiceAccount doesn’t have the permission to do it.
You should specify in your values, sidecar.rules.searchNamespace, your namespace.

Or, you disable the sidecar (sidecar.rules.enabled=false)

Rgds.