How to dynamically set X-Scope-OrgID header value in Prometheus remote_write based on metric labels?

Grafana Prometheus
, ,
1

Hi all,

I am currently using Prometheus with Cortex as my remote write backend. In my setup, I need to send the X-Scope-OrgID header dynamically based on a label from the metrics being pushed.

Here’s a snippet of my current remote_write configuration:

yaml

remote_write:
  - url: http://cortex:9009/api/prom/push
    headers:
      X-Scope-OrgID: "{ { tenant } }"  # Trying to set this dynamically from a label
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "no_.*"
        action: keep

The X-Scope-OrgID value needs to be dynamically populated from a metric label (e.g., tenant), but I am getting the following error when Prometheus starts:

ts=2024-10-15T07:36:46.819Z caller=main.go:549 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n  line 46: cannot unmarshal !!map into string"

I realize that Prometheus doesn’t support templating or dynamic variables natively in the remote_write configuration. My goal is to have the X-Scope-OrgID header reflect a value from the metric labels without hardcoding a static value.

My Questions:

  1. Is there a way to dynamically set the X-Scope-OrgID header in Prometheus based on the value of a label from the metrics?
  2. If not, are there any recommended workarounds or external tools (such as a reverse proxy or sidecar) that can help modify the X-Scope-OrgID header dynamically before the request is sent to Cortex?
  3. Has anyone successfully implemented this setup, and if so, what approach did you take?

Thanks for any insights or guidance!