Cardinality and alerts

Hi,
I wonder if someone can help me with this problem:
I have Promtail, Loki, Grafana and Alertmanager (latter as a Grafana notification channel). I need the alerts to say something meaningful about the error that triggered them - ideally the alerts should contain the stacktrace from the error.

I think I made a big mistake and used Grafana to export the stacktrace with a “regexp” filter. Now I understand that it created labels and it pushed up the cardinality.

I need help with two questions:

  • how to lower the already high cardinality without losing logs? I deleted lots of labeling rules from Promtail and Grafana but the cardinality keeps increasing.
  • how to add text from the log line (or multiline log) into the Alertmanager alert without causing cardinality issues?

Many thanks.