Hi — looking for a second opinion on Loki 3.x + OTLP (OTEL → Loki /otlp).
APIM gateway logs: some fields are already resource attrs / index labels (service.name, req.api_scp, auth.res_ag). Others are log record attrs only (req.api_id, resp.api_res_cd) — visible in Explore but not in {selectors}.
Dashboards use {service_name=“gateway-service”} + pipeline filters and scan hundreds of MB per panel.
15m cardinality: • req.api_id: 9 values • resp.api_res_cd: 4 values (200, 400, 404, 503) • Combined pairs: ~19 streams (max 36) • req_client_ori_dom: 677+ → not indexing • auth_res_affl_id: 1000+ → not indexing
Plan: promote req.api_id + resp.api_res_cd to resource attrs, add to default_resource_attributes_as_index_labels, rewrite dashboards to {req_api_id=“…”}.
Questions:
- Is indexing dynamic attrs at this cardinality (~9 + ~4) aligned with best practice, or should we stick to structured metadata?
- OTEL promote log→resource vs fixing at the OTLP source — any preference?
- Any red flags combining two dynamic index labels (~19–36 streams) on a busy tenant?