Hi,
we are running a distributed loki in our kubernetes cluster.
As of recent we are facing an issue with the query of logs.
We received logs but can’t find the logs with a search criteria that should match it, however using a different query we can find the log.
Both of the logs we are searching for have the following labels;
job: logstash-kafka
kafka_source_topic: InspectMobility-AbgeschlosseneLeistung
This Query returns only one of the logs (the first one we are searching for):
{kafka_source_topic="InspectMobility-AbgeschlosseneLeistung"}
This Query returns only returns the other log (which is obvious since we are specifically filtering for “LeistungDokumenteErstellt” which is part of the content)
{kafka_source_topic=~".+"} |= "LeistungDokumenteErstellt"
Both Logs have the same labels, so the first Query should be able to find both logs, the same time range was used for both search attempts, the line limit is 1000.
There are some other attempts at retrieving both logs, which just caused more confusion.
This one returns both logs;
{job="logstash-kafka", kafka_source_topic=~"InspectMobility-AbgeschlosseneLeistun.+"}
while this one doesn’t and returns only the first log;
{job="logstash-kafka", kafka_source_topic=~"InspectMobility-AbgeschlosseneLeistung"}
But the labels are identical for both of the logs:
Any ideas why this is happening?
I can provide additional context if needed.