Environment variables in rules provisioning

Grafana Alerting
,
1

Hi all, I am trying to setup an alerting stack via provisioning, using environment variables inside rules definition.

My environment is built on a docker stack as follows

  • grafana (v.9.4.3)
  • prometheus
  • loki
  • cadvisor
  • node-exporter

The problem is setting up an environment variables on provisioning alerting system in grafana.
I set up provisioning for alert rules, contact points, alert templates, notification policies and also datasources and all works like a charm.

I also have an environment variables on contact points provision file (relevant part only) and it works (I set MY_URL=http://foobar in an .env file passed to docker-compose)

apiVersion: 1
contactPoints:
  - receivers:
      - settings:
          url: $MY_URL

But if I do a similar thing on a rule provisioning, it didn’t work (the env variables is not replaced by current value also setted in the same .env file).
I have tried both of the following
Only a part of expr rule

apiVersion: 1
groups:
  - rules:
      - data:
        - model:
          expr: absent(node_filesystem_size_bytes{mountpoint="$MY_MOUNTPOINT"})

Entire expr rule

apiVersion: 1
groups:
  - rules:
      - data:
        - model:
          expr: $MY_EXPR_RULE

In both cases when I login to my grafana instance and go to alert rules, the query has literally the string $MY_EXPR_RULE and obviously the expression cannot be resolved.

Is it possible to use an environment variables inside an alerting rule provisioning?
Or I have to search for alternative solutions?

Thanks in advance.

2

hi @maurogiacomini

I reached out to our team internally and it seems this is the code that’s causing the problem

It looks like it was intentionally done in the rule model so as not to mistakenly interpolate macros that need to stay as-in. Such as $__timeFilter

func (queryV1 *QueryV1) mapToModel() (models.AlertQuery, error) {
    // In order to get the model into the format we need,
    // we marshal it back to json and unmarshal it again
    // in json.RawMessage. We do this as we cannot use
    // json.RawMessage with a yaml files and have to use
    // JSONValue that supports both, json and yaml.
    //
    // We have to use the Raw field here, as Value would
    // try to interpolate macros like `$__timeFilter`, resulting
    // in missing macros in the SQL queries as they would be
    // replaced by an empty string.
    encoded, err := json.Marshal(queryV1.Model.Raw)
    if err != nil {
        return models.AlertQuery{}, err
    }
    var rawMessage json.RawMessage
    err = json.Unmarshal(encoded, &rawMessage)
    if err != nil {
        return models.AlertQuery{}, err
    }
    return models.AlertQuery{
        RefID:             queryV1.RefID.Value(),
        QueryType:         queryV1.QueryType.Value(),
        DatasourceUID:     queryV1.DatasourceUID.Value(),
        RelativeTimeRange: queryV1.RelativeTimeRange,
        Model:             rawMessage,
    }, nil
}

So, we support variable expansion inside alert rule provisioning, just not inside the model

3

Hi, thanks for the explanation.

There is planning to support such a “feature”, maybe a double dollar sign $$ to treat the following expression as an environment variable, only in them rule model?
$$MY_ENV_VAR → interpolate and do env var subsitution
$__timeFiler → stay as-is

That would be a useful improvement, just my 2 cents :wink:

4

You are welcome @maurogiacomini

That sounds to me like a very nice feature to have

You are welcome to open a Github discussion about this.