How to add error log message to alert manager output alert in grafana cloud?

Grafana Loki
1

Hello,
We are using Grafana Cloud and Prometheus Alert Manager for our apps logs…

The alert rule in Grafana Cloud is somthing like that:

alert: Staging_Microservices_Error
expr: count_over_time({host="microservices", env="staging"} |= "error"[1m])
annotations:
  expr: ({host="microservices", env="staging"} |= "error")
  details: $value

Expected to get error message in the alert to slack but insted of it i recieve only values from promtail agent about error location (file, host, env).

But nobody is going to go every time to the grafana and enter query to get error message manually.

So, the question is how to get the log row with error message exactly in Gragana and send it to slack? Not just a sum of errors in log but exactly the error message is needed. Is there an approach how to implement it? I have been searching through prometheus and Grafana documentation but did not find any useful related with this case. Only found how to calculate the rows, errors and so on. Nothing related to our use case.
Please help. Thanks in advance. :slight_smile:

2

Hi,

Grafana 8 ships next week with massive improvements to alerting, including support for template variables in messages :+1:

3

Thanks but not sure that it is that i am looking for.
I need something like this for newrelik: Create an Alert from this - How to send a Notification including Log line contents - Alerts - New Relic Explorers Hub
I still can not understand why it was not implemented for grafana?
it is so hard just to implement addition of the error message to alert?
It is the basic stuff which shoukd be implemented at the beggining, but not just yet on in a future.

1 Like
4

check out Unified Alerting:

docker run -p 3000:3000 --name=grafana -e "GF_FEATURE_TOGGLES_ENABLE=ngalert" grafana/grafana:8.0.0

5

Is there a plan to support a “classic” (non-Cortex) alert manager?

6

Thanks, but we are using cloud grafana version.
Please advise how to do errors capturing there?

7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.