Extract data from logs and include them in the email alert

Grafana Loki
1

In Grafana cloud, using Loki, I want to send email alerts whenever I have the word “exception” in the logs, following my query:

rate({namespace="data", cluster="222111", job=~"data/airflow.*"} |~ (?i)exception [24h])

when running the query, this is the result I am getting:

image
image1321×707 66.9 KB

In the email alert, I want to include these info, something like this:

image
image594×707 19.2 KB

So what I have done is, creating a template which would include all of these info as follows:

{{ define "logAlerts" }}
Labels:
alertname: {{ .Labels.rulename}}
Logs: {{ .Labels.Logs }}
grafana_folder: {{ .Labels.grafana_folder }}
exception: {{ .Labels.exception }}
date: {{ .Labels.date }}
time: {{ .Labels.time }}
cluster: {{ .Labels.cluster }}
container: {{ .Labels.container }}
filename: {{ .Labels.filename }}
job: {{ .Labels.job }}
namespace: {{ .Labels.namespace }}
node: {{ .Labels.node }}
pod: {{ .Labels.pod }}
service_name: {{ .Labels.service_name }}
stream: {{ .Labels.stream }}
{{ end }}

But I see that I’m not getting any of these data, only the following:
image

is there limitations in Loki to extract these data form the logs?
if not, how we can extract info from the log line?

2

Make sure your query result actually has all those labels that you have in your alert template.

If you can provide a couple of example logs that would be helpful as well.

3

Hi,
thanks for your reply.
Yes, I have these info (except for date, time, and exception) as a result of the query, following some examples:

{cluster="222111", container="spark", filename="data.log", job="airflow123", namespace="services", node="aks", pod="driver", service_name="spark", stream="stdout"}
{cluster="222111", container="spark", filename="spark.log", job="airflow-wi", namespace="services", node="aks", pod="driver-wi", service_name="spark", stream="stdout"}
{cluster="222111", container="spark", filename="abcd.log", job="airflow-3y", namespace="services", node="aks", pod="driver-3y", service_name="spark", stream="stdout"}
{cluster="222111", container="spark", filename="airflow-yg.log", job="airflow-yg", namespace="services", node="aks-l21", pod="driver-yg", service_name="spark", stream="stdout"}
{cluster="222111", container="spark", filename="airflow-n8.log", job="airflow-n8", namespace="services", node="aks-d11", pod="airflow-n8", service_name="spark", stream="stdout"}

The exception, date, and time that I want to extract is from the log line itself. Following an example of the log line I want to extract the exception, date, and time:

> 2024-04-07 15:39:43.009 org.apache.spark.sql.delta.DeltaAnalysisException: Table schema is not set. Write data into it or use CREATE TABLE to set the schema.

I tried parsing the log line into a pattern. Following the query where I have used the pattern:

rate({namespace=“data”, cluster=“222111”, job=~“data/airflow.*”}|~ (?i)exception | pattern <date> <time> <_> <exception> [48h])

Following the output of this query:

{cluster="222111", container="spark", date=" at", filename="data.log", job="airflow123", namespace="services", node="aks", pod="driver", service_name="spark", stream="stdout", time="org.apache.spark.sql.delta.DeltaErrors$.schemaNotSetException(DeltaErrors.scala:2681)"}

Meaning, the date parameter value is " at" and not the date, time parameter value is exception.

The aim is:
time= 15:39:43.009
date= 2024-04-07
exception= org.apache.spark.sql.delta.DeltaAnalysisException: Table schema is not set. Write data into it or use CREATE TABLE to set the schema.

How to fix the query so that date, time, and exception would be part of the {} object?

4

You would aggregate by it. For example (mock query):

sum by (date,time,exception,<any other label or annotation>) (
  {SELECTOR} | QUERY
)

See Metric queries | Grafana Loki documentation