In Grafana Cloud, I setup an alert rule to send me an email when any errors are logged in Loki, for example: sum by(level, service_name) (rate({service_name="myservice", level="error"} [$__auto])).
So the email alerts include the level and service_name tags, but this isn’t very helpful to identify what the problem actually is. Is there some way to include the log message itself in there?
This query calculates the rate of logs per second, filtered by service_name and log level. The result is a numeric value (e.g., 10.4), which is then compared against a threshold (alert condition) to trigger the alert.
Including individual log messages in the alert isn’t practical because each log typically has different text, and fetching the log content could significantly impact query performance. For this reason, it’s uncommon to include the full log line in such alerts.
Instead, you can use this alert to detect when there’s a high error log rate. Once triggered, you can use Grafana Explore to dive deeper into the logs for the specific time period.