Grafana notifations is not including full information on "Value" parameter

  • What Grafana version and what operating system are you using?
    Grafana 9.3.2
  • What are you trying to achieve?
    I am trying to get notified according to my alert rules when things go bad, and with full info.
  • How are you trying to achieve it?
  • I am using http_response telegraf input plugin to monitor some of my web applications, and I set alert rules based on the result_code of the http_response plugin parameter. In query A, I put influxdb query to return a value of result_code from the plugin, and in query B I used classic conditions to serve as alert condition.
  • What happened?
  • Until recent time when my web applications go down I got the alert notification in the following format:
**Firing**

Value: [ var='B0' metric='Value' labels={server=url_of_the_web_application} value=4 ]
Labels:
- alertname = 'alertname'
- grafana_folder = 'folder'
- type = system_status
Annotations:
- description = description
Source: http://grafana.staged-by-discourse.com/alerting/grafana/YkofxuR4z/view
Silence: http://grafana.staged-by-discourse.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3Dalertname&matcher=grafana_folder%3Dalertname&matcher=type%3Dsystem_status

As you can see above the value key got many details including the URL of the web application. But presently I am not getting those details instead am just getting this:

**Firing**

Value: B=4
Labels:
 - alertname = alertname
 - grafana_folder = foldername
 - type = system_status
Annotations:
--------------------------------
--------------------------------
  • What did you expect to happen?
    I was expecting to get notified with details of the info just like before, I haven’t changed anything by my side, maybe grafana update would change that?
  • Can you copy/paste the configuration(s) that you are having problems with?
from(bucket: "system_stat")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "http_response")
  |> filter(fn: (r) => r["_field"] == "result_code")
  |> filter(fn: (r) => r["host"] == "grafana")
  |> filter(fn: (r) => r["method"] == "GET")
  |> group(columns: ["server"])
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "last")
  
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Nope
  • Did you follow any online instructions? If so, what is the URL?
    Nope

Welcome @elhananjair

Did you read thru this link? Documentation of Alert Templating in Serious Need of Improvement! - #51 by georgerobinson

1 Like

@grant2 actually in my case am not interested in customizing the notifications message content, I just wanted to know why on earth the content of Value has been changed on a recent version of Grafana, using templates in grafana is a headache as far as I know.

1 Like

@elhananjair

Before v9.3.2, what version were you running?

Also, tons of changes in the Alerting area were made on recent versions. See here.

@grant2

Before v9.3.2, what version were you running?

v9.3.0 I think

Your remarks above are the subject of many recent discussions / threads on this forum. I think your upgrading did affect your alerting content, and now you can either try to make a template, or wait until v9.4 and see if things getter better/easier.

Sorry that I cannot be of more help on this.

1 Like

@grant2 ok I will try making a template and if it doesn’t go well I will wait for v9.4
Thank you so much.

@grant2 Hello there again, I just heard about Grafana OnCall, can I shift to that for alerting system?

@elhananjair

I do not think that will help here. As I understand it, Grafana OnCall is strictly the do grouping, routing and escalation of your alerts per the diagram below. Your alerts will still contain whatever your template specifies.

1 Like