Hi! I have an alert on long running Celery tasks. However, some task can be much long running then other ones. Therefore, I would like to have different alert condition per task label - is it somehow possible? I would like to avoid duplicating the whole alert per each task. Thanks in advance.
This is not supported out of the box but you can leverage multiple data sources.
- Create a second data source, it can be a “Test data” in CSV mode.
- Populate CSV with data: each row is a metric, columns - are labels + value column. Populate the table the way that every row uniquely identifies the metric in Prometheus query response (the set of labels and their values should match). Value is your threshold
- Create Math expression something like `$filtered_duration > $thresholds
Downside of this approach is that the new dimensions that appear in Prometheus response will be ignored unless you add a row to CSV that matches it.
Same approach can be used with other data source type. Only one requirement: responses of two data sources should match by labels.