I wonder if someone can help me with this problem:
I have Promtail, Loki, Grafana and Alertmanager (latter as a Grafana notification channel). I need the alerts to say something meaningful about the error that triggered them - ideally the alerts should contain the stacktrace from the error.
I think I made a big mistake and used Grafana to export the stacktrace with a “regexp” filter. Now I understand that it created labels and it pushed up the cardinality.
I need help with two questions:
- how to lower the already high cardinality without losing logs? I deleted lots of labeling rules from Promtail and Grafana but the cardinality keeps increasing.
- how to add text from the log line (or multiline log) into the Alertmanager alert without causing cardinality issues?