Is it possible to extract fields that cannot be labelled (due to high cardinality) into alert notifications?


I understand that we’re not supposed to create labels for fields with high cardinality (e.g. source IP, or timestamps etc.);

Is it possible to somehow extract the information at alerting time and include the information in the alert notifications?

E.g. say the log looks like this
Jan 01 00:00:00 2024 level=“error”, source_ip=“”, event=“auth failure”

in the alert we have
expr: count_over_time({job=xxx} |= auth failure [5m]) > 0
event: “Authentication Failure”
message: {{$labels.event}} from IP {{$???}}

is there a way to extract the “” into the message ({{$???}} portion), and send it to the contact point as part of the annotation or something, so that I can see the IP in the alert?

Thank you very much!