Can someone confirm if this is the default behavior when using Grafana Explore to filter Loki data:
I noticed that when drilling down I see app values that do not correspond to the namespace I’ve selected.
I was not expecting to see app values that do not, and have never, existed within a filtered namespace. So I should see only one app value ingress-nginx in the namespace of ingress-nginx.
App/Chart versions:
Grafana: 11.1.5/8.5.0
Loki: 2.8.0/5.0.0
Promtail: 2.8.3/6.15.0
Windows 10 Enterprise
Docker Desktop 4.36.0
Minikube 1.34.0
I don’t see this behavior.
Can you actually run a query on {app="grafana"} for example and see what labels are present there?
I should elaborate. I only see this behavior after some time, initially it works as I expected. That is to say that when I drill down from namespace and select apps I see only apps that exist in that namespace. Here is a set of working and not working requests:
Working:
api/datasources/uid/P8E80F9AEF21F6940/resources/label/app/values?start=1734619845053000000&end=1734623445055000000&query=%7Bnamespace%3D%22ingress-nginx%22%7D
{
“status”: “success”,
“data”: [
“ingress-nginx”,
“ingress-nginx”,
“ingress-nginx”
]
}
Not Working:
api/datasources/uid/P8E80F9AEF21F6940/resources/label/app/values?start=1734621031219000000&end=1734624631220000000&query=%7Bnamespace%3D%22ingress-nginx%22%7D
{
“status”: “success”,
“data”: [
“ingress-nginx”,
“kube-proxy”,
“minikube”,
“promtail”,
“storage-provisioner”
]
}
Also, I played around with the chunk_idle_period setting and it seems to be consistent that the mismatch behavior occurs after double the amount of this time. For example, for brevity, you can set this to 2m30s (as I experimented with) and you’ll notice after 5 mins this mismatch (and prior you see things work fine).
One other thing I noticed is that no matter what the Label Browser always works consistently:
Here is the query on just the app Grafana:
