Request to modify email template for grafana9.1.7

Hello,

I’m using Grafana v9.1.7 (0cbb79298d) and trying to use email template for the alerts.

As an example, we have an alert set up for the panel and the legend is set up like this.
{{hostname}} - {{container_name}}:{{log}}

The email alerts from this panel respect whatever we mark as the legend in the previous Grafana version, however for our new Grafana instance we don’t get this behaviour. No matter what we mark as the legend for the alert panel the emails come through in the mixed and complex format.

The custom template we’ve tried to use and apply is as for example, please help us to formulate the message template -

{{ define "my_template" }}
  .Message = {{.Message}}
  .Metric = {{.Metric}}
{{ end }}

Thanks,
Eugene

Hi! You need to first range over the alerts, as notification templates work with lists of alerts, not individual alerts. I’d recommend checking out the documentation here, including Using Go’s templating language. If you still have questions after reading through these docs I’d be happy to answer them!

Hi George,

Have responded over the direct message and would appreciate if you could assist on this particular request.

Thanks!
Eugene

Hi George,

As discussed, please find the details as follows and would appreciate if you could help to formulate the template.

The output information in the current email alert is quite explicit and contains lots of extra details, thus would like the notification email to look in a more readable format.

In particular, we would only want to get information of the log argument out of all the rest info as shown on the screenshot attached (could be also helpful to have the instance/hostname and the container_name arguments).

Please let us know if you need more information.

Thanks,
Eugene

I think you want something very similar to what I suggested here. I’ve changed it below to better match your situation.

  1. Instead of using Classic Conditions change the alert rule to use Reduce expression and then either a Threshold or Math expression. This will allow you to use $labels in both annotations and notification templates.

  2. Add a summary description to your alert rule with the following template:

{{ index $labels "log" }}
  1. Create a custom notification template like this and use it in your contact point:
{{ define "alert.summarize" -}}
{{ index .Labels "alertname" }}: {{ index .Annotations "summary" }}
{{- end }}

{{ define "alerts.summarize" -}}
{{ if .Alerts.Firing }}{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}

{{ if .Alerts.Resolved }}{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}
{{ end }}

Thanks for your response. I’ve updated the alert rule to use the Reduce expression though the email alert was received in the following format. -

Please find the rest of the steps are set as follows.

  1. Changed the alert rule to use the Reduce expression (please let me know if it needs to be updated as well).

  2. Added the summary description to the alert rule. -

  3. Created the custom notification template.

  4. Updated the contact point.

You are using Loki, right? Do you want to alert if count_over_time is more than 0, or is it some other number? If its 0, you need a Math expression, just a Reduce expression. If its another number, like 5, then the Math expression should be $B > 5.

If possible please create some logs that match this query so we can see what labels are returned by the query. I see there is no data, which makes it more difficult to troubleshoot.

Hi George,

Yes, we’re using loki for logging and we would want the alert to be triggered only if count_over_time is more than 0 (the condition is fine if the count_over_time equals 0).

Please find the logs and labels as follows -
{container_name=~".*<app_name>.*", env="prd"} |~ "(?i)parameters" | json | line_format "{{.log}}"

Please let us know if more information is needed.

Thanks,
Eugene

I suspect the issue here is that log is not being returned as a label from Loki. Here I have an example of a Loki rule I created that matches test1="hello", and it returns two labels: test1 and test2:

Do you see log as a label like I see test1 and test2?

Hi George,

Please find the query I’ve used and let me know if the condition needs to be updated as well in order to displayed the data properly -

Thanks,
Eugene

I think there has been some confusion here :slight_smile: Please use your original Loki query but show me the result of the Preview button like in my example. I want to see what labels are returned by your query. You might need to make sure the query returns data if it returns No Data.

It appears that log is not returned as a label in fact though please correct if the steps are accurate and/or if needed to update the alert configuration -

Thanks! If you change the expression B from a Classic Condition to a Reduce and click Preview, is it still not a label?

In the meanwhile, I can ask the Loki team because I suspect there is something we need to change in the query for this to work.

Looks like it’s part of the Info column in the Preview, please find the details as follows.

[ var='B' labels={attrs_tag=<app_link>:release-v1.0.0-106|<app>|sha256:2b62f8a300ddf1af70832678312c58a4c397f1b74631d2fe43c9f6fdb65ecd46|a46c0618d259153766d25217e56845e30aa1597b398c7ff48c958a765d70546c, container_id=a46c0618d259153766d25217e56845e30aa1597b398c7ff48c958a765d70546c, container_name=<app>, env=prd, filename=/var/lib/docker/containers/a46c0618d259153766d25217e56845e30aa1597b398c7ff48c958a765d70546c/a46c0618d259153766d25217e56845e30aa1597b398c7ff48c958a765d70546c-json.log, hostname=<instance>, image_id=sha256:2b62f8a300ddf1af70832678312c58a4c397f1b74631d2fe43c9f6fdb65ecd46, image_name=<app_link>:release-v1.0.0-106, instance=<instance>, job=containerlogs, log=on=2023-03-23T14:50:01 at=INFO module=<module_name> CID=532270ad-6646-40be-9c69-064f0d47531d msg="Request parameters: {"Select": "TradeId,DealId,Action,EffectiveDate,<Group>,Folder,InstrumentName,Quantity,Direction,<Product>,<ProductType>,TradeCcy,CurrencyPair,Premium,PremiumPaymentCurrency,SettlementType,CallPutIndicator,Valuation,Barrier,BarrierStart,BarrierEnd,BarrierDown,BarrierMonitoring,SettlementDate,FarSideSettlementDate,ExpiryDate,FixingDate,FixingTime,FixingCity,PaymentAt,Price,StrikePrice,UnderlyingInstrumentName,ValueDate,Comments", "Where": "((UpdateTime>=DateTime(2023,3,23,14,0,5,\"Utc\") and UpdateTime<=DateTime(2023,3,23,15,0,0,\"Utc\") and <ProductType>.Contains(\"<Product>\") and SettlementDate>=DateTime(2023,3,23) and (Folder=\"TARUN_1\" or Folder=\"H278A\" or Folder=\"H278C\" or Folder=\"H278D\" or Folder=\"H278E\" or Folder=\"H278B\")))", "GroupBy": ""}" , stream=stderr, stream_extracted=stderr, tag=<app_link>:release-v1.0.0-106|<app>-1|sha256:2b62f8a300ddf1af70832678312c58a4c397f1b74631d2fe43c9f6fdb65ecd46|a46c0618d259153766d25217e56845e30aa1597b398c7ff48c958a765d70546c, time=2023-03-23T14:50:01.510273313Z} value=1 ]

Hi @georgerobinson , just a gentle reminder please if there’s an update.

Thanks!

Thank you George for the help and response, replying to the main topic thread.

Appreciate if you could help to limit the current custom template to only have the Labels fields and help to get rid of the summary and Value fields (we would only prefer to keep the Labels field).

Please let me know should we also update the custom message template for this accordingly?

{{ define "alert.summarize" -}}
{{ index .Labels "alertname" }}: {{ index .Annotations "summary" }}
{{- end }}

{{ define "alerts.summarize" -}}
{{ if .Alerts.Firing }}{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}

{{ if .Alerts.Resolved }}{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}
{{ end }}

Thanks,
Eugene