Navigating from an alert notification to a time constrained log dashboard

[amueller2]

Hi all,

I have an alert with the following Loki query:

count_over_time({compose_project=~"project1|project2"} |= `ERROR` | pattern "<log_timestamp> <log_message>" [1m])

For this, I created a simple description template:

Timestamp: {{ $labels.log_timestamp }}
Message:  {{ $labels.log_message }}

What I want to add is a href to a log dashboard which filters it to basically from= and to=<timestamp of the log + 1 minute> - so that a recipient of the alert can click it to get the full trace of the log messages at that time without having to manually filter the log dashboard down to the actual date range of the error(s).

I tried various approaches without success, basically, the blocking point seems to be that my timestamps are in 2025-11-17T16:32:52.236+01:00 format and I didn’t find any way to convert them to unix nanoseconds in order to do the + 1minute calculation and construct the href inside the template.

Or is there any other approach I could take here? Any ideas are appreciated!

[pepecano]

Hi @amueller2,

This is often done using a custom notification template that uses the StartsAt or EndsAt fields of the alert object. See the example Print a link to a dashboard with time range:

{{ define "custom.my_dashboard_url_annotation" -}}
{{ range .Alerts -}}

  {{/* StartsAt - 1h */}}
  {{- $from := (.StartsAt.Add -3600000000000).UnixMilli }}

  {{- $to := "" }}
  {{- if eq .Status "resolved" }}
     {{- $to = (.EndsAt).UnixMilli }}
  {{- else -}}
    {{/* Use current time if alert is firing */}}
    {{- $to = (time.Now).UnixMilli }}
  {{- end -}}

  Dashboard: {{.Annotations.MyDashboardURL}}?from={{$from}}&to={{$to}}
{{ end }}
{{ end }}

[amueller2]

Thanks! I’m afraid I still don’t grasp the difference between “Template annotations” and “Template notifications” in Template annotations and labels | Grafana documentation

In the first diagram in that page, a “Notification template” (I assume this is the same as “Template notifications”) is tied to a specific contact - which is not the right place for this kind of template because it should be independent from the contact point and not needed to be copy&pasted for all contacts?

So that leaves “Template annotations”, is this meant to be the alert’s “6. Configure notification message” → Add Custom annotation with a name and a value, but since you took the example from section Template Notifications, I assume I can’t just put the template into a custom annotations value?

Still slightly confused…

[pepecano]

@amueller2

I’m afraid I still don’t grasp the difference between

@amueller2, you’re definitely not the first one to run into this. In short, there are two types of templates that use the same template language but differ mainly in the template context/data and the available functions. The docs go into the details:

• Template annotations and labels
• Template notifications

@amueller2

In the first diagram in that page, a “Notification template” (I assume this is the same as “Template notifications”) is tied to a specific contact - which is not the right place for this kind of template because it should be independent from the contact point and not needed to be copy&pasted for all contacts?

A single notification template can be reused across multiple contact points, so you don’t need to copy and paste it per contact point.

The notification preview feature is handy when building and testing your custom templates.

Once you create a notification template, it will show up in the Select notification template list when you assign a template to a specific contact point field.

Hope this helps!

[amueller2]

Great, I think it finally clicked! The only minor issue now is that I can’t seem to generate a real clickable href, with the link is just text in an email, with

<a href="{{.Annotations.MyDashboardURL}}?from={{$from}}&to={{$to}}">Dashboard</a>

that string is rendered verbose and not as actual href.

[pepecano]

@amueller2

The only minor issue now is that I can’t seem to generate a real clickable href,

Yes, it’s known issue. From docs:

You can customize the email subject and main section of the email body, but you can’t edit HTML or CSS for visual changes.`