We’re using the
loki-distributed helm chart in GKE and having issues with the ingester spamming us with “failed to flush user” errors that are related to IAM issues:
Post “https://storage.googleapis.com/upload/storage/v1/b/…”: compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\thttps://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`
This would suggest that the ingester is perhaps using the default service account instead of the one configured on the pod. I’ve verified that the service account on the pod should have the right permissions, and no other service appears to be complaining. One mitigation would be to set up our cluster’s node config so that
storage_rw is included in the default OAuth scopes, but we’d rather stick with using workload identity.
Am I missing something? Configured something wrong? Let me know if more info would help.