I have a Grafana alert rule defined, which monitors my jira service logs in Azure.
The alert fires when it is supposed to fire and sends the message to the proper notification channel.
My problem is that I am not able to extract the problematic logs from Azure and inject them into the alert message sent by Grafana.
I have tried a couple of things but nothing worked so far.
My latest version of the alert definition looks like below. BTW this version is not working in Grafana (Becaue of the message being added to the summarize section), but it gives you an idea of what I want to achieve.
Thanks
- alert: errorUnexpected
for: 1m
subscription: "/subscriptions/SUBSCRIPTION"
query_type: "Azure Log Analytics"
azure_log_analytics:
query: |
ContainerLogV2
| where $__timeFilter(TimeGenerated)
| where ContainerId in (
KubePodInventory
| where TenantId == "ID"
| where ServiceName == "jira"
| project ContainerId = ContainerID
)
| where LogLevel in ("error", "unknown")
| where tostring(LogMessage) matches regex @"(?i)Unexpected error"
| extend message = replace_string(substring(tostring(LogMessage), 0, 300), "\n", " ")
| summarize EventsPerMinute = count() by bin(TimeGenerated, 1m), message
| order by TimeGenerated asc
resources:
- /subscriptions/SUBSCRIPTION
threshold: 0
labels:
disabled_resolved: true
severity: critical
category: logs
message: "{{ if $labels.message }}{{ $labels.message }}{{ end }}"
annotations:
description: "Unexpected error in jira service: {{ $labels.message }}"
summary: "Unexpected error in jira service"