Hey y’all, is we just got Prometheus metrics from Loki and Promtail, but metrics like promtail_read_bytes_total only contain the read bytes for a particular Promtail instance. We’d like to sum these up by Kubernetes namespace and the app label on our application similar to we can query logs for our apps with LogQL {namespace="my-ns"}
I’m not sure if that’s possible to configure though. The reason why we’d like these metrics to be in Prometheus is that our Loki instances failed a few times due to filled up volumes when some pods start logging way too much. We’d like to be able to identify which of our deployments is logging too much.
When the storage gets full, we always experience data loss due to corrupted files in our volume.
What would be the best option to get access to metrics on information like this?