Ingester not using configured service account for storage

[gedge]

We’re using the loki-distributed helm chart in GKE and having issues with the ingester spamming us with “failed to flush user” errors that are related to IAM issues:

Post “https://storage.googleapis.com/upload/storage/v1/b/…”: compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\tUse Workload Identity  |  Google Kubernetes Engine (GKE)  |  Google Cloud

This would suggest that the ingester is perhaps using the default service account instead of the one configured on the pod. I’ve verified that the service account on the pod should have the right permissions, and no other service appears to be complaining. One mitigation would be to set up our cluster’s node config so that storage_rw is included in the default OAuth scopes, but we’d rather stick with using workload identity.

Am I missing something? Configured something wrong? Let me know if more info would help.

[gedge]

I think I perhaps misunderstood how this works. I believe I need to get a key and have a GOOGLE_APPLICATION_CREDENTIALS env pointing to it. Going to try that.

[gedge]

Finally figured out my issue: we had our IAM membership set up so that the service account name had to be loki-app-staging, and my tweaking the full name value in the helm chart resulted in the service account being loki. I’m new to all of this IAM / workload identity stuff and couldn’t make the connection. Everything’s working now