Missing labels in the Loki explore label filter and label browser

Hello, if I try to create a Loki explore query and I want to select a label, the UI dropdown only displays a subset of the available labels.

Even though I can see (via the browser developer tools) that all the labels are returned in the api/datasources/uid/.../resources/labels HTTP query.

The exact same behavior can be seen when using the Label Browser.


What could be the issue here?

The labels were being properly displayed until about a week ago.

FYI: We are using Grafana Cloud.

  1. Do the logs show those labels when querying in the explorer?
  2. What if you put in a manual query such as {__aws_cloudwatch_owner=~",+"}, does it return anything? Does it return from API?

I try not to use any label with leading __. You might consider removing those when sending logs to Loki and see if that helps.

  1. Yes the logs contain those labels. If I manually input the query, then there is no issue.

  2. Yes, as you can see on the first image. The query functionality is not impacted and the logs contain the appropriate labels. Only the Grafana UI is impacted.

Interesting. I am not sure, haven’t seen this before, hopefully others can chime in.

What changed between when it was working to now? If you are using Grafana Cloud you might also consider reaching out to support.