Hi! Your custom template is printing the ValueString, which will always look like your original screenshot. If you want to do something different you can instead iterate over $values. We have an example of that in the new default template alerting/default_template.go at main · grafana/alerting · GitHub
Hi! This is a special kind of alert called DatasourceError. It means Grafana was not able to query your datasource to evaluate the alert. This can happen if the datasource is misconfigured, unreachable from Grafana, or was down at the time this alert fired.
Hi! Thanks for reply. Sounds strange, as we have everything in k8s and if there something is down, we get repots and alerts. But, if alert is on thanos datasource, and alerting from grafana, how this cam be fixed? Need to fix all alerts?
Hi! I would first check that your Grafana server can talk to your Thanos datasource. When there is a DatasourceError alert, the reason for the error will be in the annotations.
OK! If this continues to occur it’s best to check the Error annotation as this will tell you exactly why Grafana was unable to query your Thanos datasource.
hey
i need to access the threshold value i set to use in in the message
i mean if i set the threshold to something, i will be able to use it as a variable in the message and also play with it. in a way that using an if clause, change the message accordingly. so when it fires the message say that the value X is greater than set threshold of Y and in the resolve message i get the OK value X is less than the set threshold of Y. or something like this.
Hi! I have been testing Your suggestions, there are no error. I managed to sent alerts to different channel, but with new default template and it looks like this:
As You can see, there is no error with Data, or datasource. Grafana sends out alert with values, but with default template, it is empty. And there is most useful information from that alert. Is there anyway to fix Value and sort them, or it is can not be done at the moment?
Forgot to mention, that screenshots are not from full alert, rest of it is ok. This is all about Value
Hi! Your original screenshot was a DatasourceError alert as there was an alertname label DatasourceError, but you are indeed correct that your most recent screenshots are not DatasourceError alerts.
If you would like to iterate over each of the values, including the labels, then something like this might work:
{{ range .Alerts }}
{{ range $k, $v := .Values }}
{{ $k }} = {{ $v.Labels }} {{ $v.Value }}
{{ end }}
{{ end }}
All examples appear to be such that merrily iterate over all alerts and related labels.
Simple example on how to pick a subset of labels or one certain label to print is either very hard to recognize or does not exist.
The defaults with just few triggering alerts tend to be counterproductive when delivered to chat, taking more than screenful of space. Combine that with slowly flapping alert and the alert itself is an problem.
@georgerobinson Hello, I’m using Grafana 9.5. I have two questions about alert notifications, hope you can help.
First, involves an email subject line:
I have two alerts ( named MDS1 & MDS2) in an evaluation group, “MDS”.
I would like to configure an email notification so that the email subject line is either -
“ddn-18k-mds-eth1 15 minute load average has exceeded 100: load is [value]” or
“ddn-18k-mds-eth2 15 minute load average has exceeded 100: load is [value]”. depending on which is firing. (note that one is eth1, the other is eth2)
Value is the value returned in the alert, see shot below. Value now returned is 14 decimal places long. I only need 1 decimal place, perhaps even better only the whole number.
I want this subject line only for MDS alerts, not for any other alerts I have now, or may create in the future.
Is it possible to do the above and still get resolved subject lines: “0 firing alert(s), 1 resolved alert(s)” ?
Already been through the documentation, but cannot make out how to create this subject line. Looking for appropriate code, and where to use it i.e. alert rule annotations, or Contact Point /optional email settings, or Contact Point / notification template?
When an alert query exceeds the threshold (100 in this case) I get notification that the Value is 118, which is expected. At the next evaluation, if the value has climbed to, say, 120, another notification is issued, and so forth for as long as the value increases. How can I eliminate these additional firing notifications?
There is much more chaff in the email above that I need to clear out, but an informative subject line and cutting redundant notifications will be a great leap forward. Thank you.
I would like to configure an email notification so that the email subject line is either -
What should the subject line show if both alerts are firing at the same time?
Is it possible to do the above and still get resolved subject lines: “0 firing alert(s), 1 resolved alert(s)” ?
Yes, it does increase the complexity somewhat. What do you want the subject line to be if the notification contains both firing and resolved alerts?
When an alert query exceeds the threshold (100 in this case) I get notification that the Value is 118, which is expected. At the next evaluation, if the value has climbed to, say, 120, another notification is issued, and so forth for as long as the value increases. How can I eliminate these additional firing notifications?
You’ll want to change repeat_interval in the configuration of your notification policy.
Hi George,
I have two alerts, so I’m getting two separate emails now. Due to the nature of these particular metrics, they will not fire at the same time. One is a backup for the other.
Any way to specify decimal places? I know that can be done for actual Graf charts, but I don’t find similar, at least in Alert Rule creation, per set.
Thanks, I’ll check out the repeat_interval.
What do you have in the “Alert Rule” for this alert under #4 “Add defaults to your alert rule”.
Do you have “ddn-18k-mds-eth1” statically entered in “Summary” or it’s a value of one of the “Labels” ?
Hello @graffan I have a static message, and runbook URL in #4:
That message only appears in the body of the email. I’d like it, with load is [value] to be in the email subject line. But there is no text field for subject line.
