I’m using Unified Alerting with Grafana open source v9.1.6 self-deployed in a k8s cluster
I installed kube-prometheus, and as a result I have hundred of rules and alerts, and I would like to visualize them in our Grafana dashboards. This is what the alerting section looks like for a firing alert:
Key point here is the top 3 labels. Those do not appear in the Alertlist panel, which means I’m unable to filter by the cluster tag:
As you can see, “CPUThrottling” alert appears twice because I’m currently monitoring two clusters.
The cluster label is an “externalLabel” defined like this:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.35.0
name: k8s
namespace: monitoring
spec:
replicas: 2
image: quay.io/prometheus/prometheus:v2.39.1
thanos:
baseImage: quay.io/thanos/thanos
version: v0.28.1
objectStorageConfig:
key: thanos.yaml
name: prometheus-thanos-objstore-config
additionalScrapeConfigs:
name: prometheus-additional-scrape-configs
key: prometheus-additional.yaml
alerting:
alertmanagers:
- apiVersion: v2
name: alertmanager-main
namespace: monitoring
port: web
enableFeatures: []
externalLabels:
cluster: qa-microservices-primary-us-west1
I’m not sure if Grafana Alerting does not support this or if I’m missing something here?