Hey there!
Crossposting this from Slack here, maybe somebody here has an idea. 
I’m using the following recording rules to store the ingested data in a metric every hour to be able to calculate an average over a month in Thanos later.I’d like to record the tenant_id label, which is actually set if I run the expression directly. If I’m using it in the recording rule, tenant_id is set to “default-tenant”, which puzzles me. Any idea where that tenant id comes from?
groups:
- name: loki-ingest-1h
interval: 1h
rules:
- record: loki:ingest:sum1h
expr: sum by (tenant_id) (bytes_over_time({tenant_id=~".+"} [1h]))
I don’t see a label named tenand_id from my Loki cluster, do you perhaps have multi_tenant_queries_enabled enabled for your Loki cluster?
Anyway, I don’t think it’s possible, and even if it is I don’t think it makes sense. As far as I know Loki Ruler has tenant-specific rules, and they are evaluated against the tenant itself, so even if you had a tenant_id label you won’t see any data from other tenants. There appears to be a feature request for federated rules (Add federated rules · Issue #7659 · grafana/loki · GitHub).
I think your only option for now is to have identical rules for each tenant, and hardcode the tenant ID in the expression itself.
The tenant_id is a label I’ve added as a global label in all my ingestion clients (promtail, etc.).
But you’re right, I already have tenant-specific rules, so I simply inject the tenant_id there again.
I was just wondering where that “default-tenant” comes from.
