I know that general guideline for loki labels is fewer is better.
With this is mind, should I use pod name as loki labels?
I’m asking this because in my case, our cluster runs thoudands of short lived pods per day, each with its unique name, and small amount of log.
How could this (to use or not to use) affect performance of loki?
In this case the pod name should not be used as a label. This causes high cardinality for the labels which will slow down Loki.
You can still include it as a field in the log line. It can then be filtered on at run time. Loki was built to handle this use case so it’s not as inefficient as it appears.
Can you clarify what types of queries you want to run?
Thank you for the reply
we use queries like below to caculate errors when running workloads in kubernetes:
count_over_time(({app="test-app"} |= "ERROR")[1m])
I think this is confusing because the default behavior is to use pod name as label, so if there aren’t many shot lived pods in cluster, it’s ok to use it?
I had the same query. If you don’t use the pod name as a label.
Lets say there are multiple pods running on different machines with the same app label. Then all the logs of these pods would be appended to the same stream. As the machines are different there is a possibility that there is a few millisecond time differences between them. Then there is a chance that the logs generated would be marked as out of order by loki.
This is just my guess. Can someone confirm this?
There is this design document about ordering constraint removal. I am not 100% sure if this has been implemented or not.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.