"failed to build table names cache" posted on Loki backend pod

Grafana Loki
1

Here is the background:

Our loki stack (ssd mode, version 3.4.2, k8s deployment) is writing log files to bucket A for the past months but owning to the config change, we have to switch to a new bucket, say bucket B. The data sync between 2 buckets has been done successfully via aws s3 sync and the old log files can be queried successfully after this switch.

Today from the dashboard we found loki backend started posting errors like below:

level=error ts=2025-04-22T09:31:16.492835891Z caller=cached_client.go:189 msg=“failed to build table names cache” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:31:17.503242217Z caller=cached_client.go:189 msg=“failed to build table names cache” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:31:17.607369372Z caller=ruler.go:576 msg=“unable to list rules” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:32:17.608831269Z caller=ruler.go:576 msg=“unable to list rules” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:33:17.607800328Z caller=ruler.go:576 msg=“unable to list rules” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:34:17.607628064Z caller=ruler.go:576 msg=“unable to list rules” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:35:17.60755593Z caller=ruler.go:576 msg=“unable to list rules” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”
level=error ts=2025-04-22T09:36:16.491280885Z caller=delete_requests_table.go:92 msg=“failed to upload delete requests file” err=“NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors”

I can confirm the credential is correct and successfully mounted to Loki.

One thing I have to mention is that we turned on retention_enabled in compactor section and most of the tenant logs are set 30d retention policy. Will the bucket migration affect the consistency of the retention setting such as the index or some other tiny but critical part?

Though the issue has been fixed by reinstallation, I am still concerning about whether it will re-occur in the future.

Any thought would be helpful.

2

@tonyswumac Hi Tony, could you please help check this one if there is any availability?

3

I am not sure, if you already fixed the problem with a reinstall I don’t think it’s too concerning.

Are you using IAM user? What sort of credentials / access control do you have on your Loki cluster?

4

hi Tony,

Thanks for the reply.

Actually it is not the genuine AWS S3 but ECS storage that working as object storage. The user that I am using for storing the chunks is created locally which has full control on the bucket.

I checked with our storage guy and they found during that period, there was no request sent from Loki, thus I am afraid this should be something wrong of Loki Backend which failed to retrieve the credential from the configmap. However, I did exec into the pod and made sure the config file is correct.

Same issue happened once before, but that was owning to the configmap template wrongly render.

5

Sounds like a non-standard setup. Any reason you are not using S3 if you are already in AWS?

Also please share your configuration.

6

hi Tony,

Thanks for the reply.

We are running within the internal environment and not allowed to post the data to any public cloud like AWS, ECS is the chosen product for object storage. :frowning:

Btw, ECS supports s3 protocol and which should be agnostic to Loki I think.

7

What exactly do you mean by ECS? I am probably thinking of something else.

Also please share your configuration.

8

Hi Tony,

ECS is short for Enterprise Object Storage from Dell Technologies, it is local storage simulating the S3 protocol from AWS.

Here is the configuration in my configmap:

image
image818Ă—658 32.8 KB

9

I don’t see anything obviously wrong. And given the fact that you have it working we reasonably be able to say that it’s not a problem with Loki itself.

1 Like