Prometheus and Grafana Alerting for Absent Metric

Grafana
,
1

Hello everyone,

I’m currently working with Prometheus and Grafana for monitoring purposes, and I’ve run into a challenge with setting up alerts for the absence of a specific metric. I hope someone here can provide some guidance or insights.

I have a metric called process_resident_memory_bytes that represents the memory usage of some type, and I want to create an alert that triggers when this metric stops reporting for any organization and cluster. In Grafana, I’ve tried to use the following Prometheus query, but its work just on specific org and cluster

absent(process_resident_memory_bytes{org_id=“some_org_id”, type=“some_type”, cluster_id=“some_cluster_id”}[30m]).

and it indeed returned me 1 when the metric stopped reporting.
But, I would like to create an alert that sends a notification when this metric is absent for any combination of org_id and cluster_id not just specefic example of org id and cluster id.

Can anyone please provide guidance on how to properly set up this alert in Grafana using Prometheus queries? Any advice, suggestions, or examples would be greatly appreciated.

Thank you in advance for your help!

2

There is no option to do that with absent.

But you can do something similar with timeshift:

  • create query which will group by selected labels for metric in the past (e. g. 1 hour shift)
  • create query which will group by selected labels for current metric
  • join these 2 queries and detect which timeseries are missing somehow - those are timeseries, which have disappeared meanwhile and you can alert on them

That’s a suggestion , which can be implemented in one PromQL query.

1 Like
3

Thank you! It work perfect

4

It will be nice if you show your query for the record, pls.

5 
group by (org_id, cluster_id) (
             process_resident_memory_bytes{type="some type" } offset 1h
             unless on(org_id, cluster_id)
             process_resident_memory_bytes{type="some type"}
)==1
1 Like