Grafana v11.0.0: unable to capture key value in alerts notification

Grafana Dashboards
1

I have a Lucene query that fetches log data from Elasticsearch datasource:

kubernetes.name: $customer AND x_Duration : * AND message : *

I set up alert rule using the default Grafana template.

In alert notification email, I see these Labels keys: “alertname”, “grafana_folder” and “kubernetes.name”.

I dont remember explicitly configuring these labels, so i assume they are built in in the default notification template.

I would like to add an extra Label called “message” and display its value in the notification email.

As seen above, my query does contain field name “message” and i confirmed it is present in raw data.

Therefore i tried to accomplish this task like this:

  1. I went to panel’s Alerting menu and under “Configure labels and notifications” i created a new label with a key “message” and the value "{{ $values.A.Value }}

  2. Then I went to Contact Points - Optional Email settings and in the Message field wrote this:

{{ range .Alerts }}
Instance: {{ .Labels.instance }}
Summary: {{ .Annotations.summary }}
Values: {{ .Annotations.message}}
{{ end }}

However, in the email notification, the Values field is empty.

How do i get the contents of the “message” field to be displayed in email notification?

2

Just try to group by message field in your alert query - that should be recognized as a label by Grafana automatically.

3

ok, i grouped by message.keyword and in the notification email i see the value of this field but it is greatly truncated and appears to have some extra keyword that does not appear in my original data:

image

All the text in the above printscreen is found in my original log, but it abruptly ends with “dsqlExecutePrepared” which i have no idea where it comes from - definitely not from my logs.

After the words “performance threshold of 1.0000 seconds” in my original logs there is the actual SQL statement, sometimes as long as 500 characters, but for some reason Grafana notification does not display them and instead puts this “dsqlExecutePrepared” at every occasion

Here is how my original log data looks like (only a snippet):

Statement execution took 18.0918 seconds, exceeds performance threshold of 1.0000 seconds. ‘SELECT MAX (y_acttime) FROM (SELECT TOP 100000 y_acttime FROM dmv_audt WHERE o_acttime < ? AND o_objtype = 16 ORDER BY o_acttime ASC) a’"}

4

I can imagine that alert annotation doesn’t have unlimited value, so it can be truncated. Use query inspector and check how results are returned if they are grouped by message, if message is properly parsed/mapped on ES side,…

You are grouping on message.keyword and type keyword has some mapping config, e.g.:

{
  "foo": {
    "type" "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}

So I think that truncation is on ES side, not on the Grafana in this case.