Grafana loki MS Teams (Grafana Version 10.4.1)

[fonmedigj]

Hi community,
I am looking for assistance with setting up alerts in Grafana using loki as the datasource. My goal is to send alerts to MS Teams with the specified timestamp and message from my log line attached.
Alert Requirements:
Timestamp: Extract the timestamp from the log line.
Message: Extract the message from the log line (e.g., “NoLocation Collision detected for number +1234523330666…”).

I am using this template for my contact point
//
{{define “lokiToMsTeams” }}
{
“title”: “Error in {{.Fields.app }}({{.Fields.container }})”,
“summary”: “Error encountered running importer process”,
“sections”: [
{
“activityTitle”: “Error Details”,
“activitySubtitle”: “Error message”,
“facts”: [
{
“name”: “App”,
“value”: “{{.Fields.app }}”
},
{
“name”: “Container”,
“value”: “{{.Fields.container }}”
},
{
“name”: “Filename”,
“value”: “{{.Fields.filename }}”
},
{
“name”: “Job”,
“value”: “{{.Fields.job }}”
},
{
“name”: “Namespace”,
“value”: “{{.Fields.namespace }}”
},
{
“name”: “Node Name”,
“value”: “{{.Fields.node_name }}”
},
{
“name”: “Pod”,
“value”: “{{.Fields.pod }}”
},
{
“name”: “Stream”,
“value”: “{{.Fields.stream }}”
}
],
“text”: “{{.Line }}”
}
]
}
{{end }}
//
My alerts on Teams appear like below

Could someone guide me on how to configure this alerts to Teams with the specified timestamp and message from log line? Any examples or documentation references would be great appreciated,

Thank you in advance for your help

[yuriy.tseretyan]

The template does not look correct. The field Fields does not exist in the template context. You can learn more in the documentation Create notification templates | Grafana documentation and Notification template reference | Prometheus

Also, the alerting is label-based, meaning that all information from the rule evaluation should be in labels. Also, it means that only metric queries to Loki are supported. To see what labels your query returns, click Preview button. It will return you a table with multiple rows (we call them dimensions), every row will have a list of labels and value. Those labels will be available in the template.

[fonmedigj]

Thank you @yuriy.tseretyan for the great tips. I can now use labels to get my alert as envisioned.
How do I make the alert presentable, edit and remove these e and how do I add line breaks like
, they seems to get printed in my alerts

[yuriy.tseretyan]

It is hard to tell what is going on in the screenshot. First, you need to determine how the alert looks like, where the data is, in annotations or in labels. Then write a template in the notification. You can find default templates here. Also, you can write templates in rule definition, annotations and labels support templating.

[fonmedigj]

Thanks again @yuriy.tseretyan so this is how my alert rule looks like

Below are my labels and preview

My Contact Point message looks like this

Is there a reason my alerts don’t carry a timestamp as shown in the preview above?

[fonmedigj]

Hi @yuriy.tseretyan Please help clarify this, I’ve been reading the documentation and trying different options with no luck

[yuriy.tseretyan]

Sorry for delay. The template mentions {{CommonLabels.log_timestamp}} which may not be true if that label changes often and it’s not mentioned in group.

When the Alertmanager receives alerts (the one that you see in preview) that are identified by the set of labels, it processes them through notification policies, and then applied group_by parameter of the matching policy. The set of labels specified in group_by identify a group of alerts. That group lives in memory, and during the runtime Alertmanager can add and remove alerts from that group.
Now, every alert group is processed by a pipeline that is run every group_interval of the matching notification policy. When pipeline is run, it creates a snapshot of the group, which can contain one or many alerts. Then it calculates CommonLabels, i.e. labels have the same value for all alerts in the group.
When rule is evaluated every alert instance is uniquely identified by set of labels. Therefore, two alerts that have different values of the same label - different alerts.

Now back to the log_timestamp (and timestamp). That label seems to change every time the rule is evaluated, and therefore every time a new alert is created. However, due to group_by policy, all those alerts end up in the same group, which can cause that log_timestamp gets omitted from CommonLables.

I would recommend avoid such labels as much at possible because it can cause a lot of flapping alerts and quite useless state history.

To debug your alerts, I would recommend adding webhook integration that points to some debugging service such as webhook.site. So you can see exactly what’s going on with your alerts.

[fonmedigj]

Thank you @yuriy.tseretyan yes it make sense what you are saying. It is possible to format my alerts and add action buttons on them? This is how my alert arrives on MS Teams, and as you can see it doesn’t look presentable or readable. What options do I have to fix this?

[yuriy.tseretyan]

Look into MS Team workflows. It provides more control over the final formatting of a message