Structured metadata in loki helm chart

Grafana Loki
,
1

We’re using the simplescalable helm chart, and its working fine. I’d like to move to structured metadata for some data but cant see it even though it seems to be sent. From what i can tell in the docs, all boxes are ticked (If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.) - What is structured metadata | Grafana Loki documentation

  • alloy (clients) v1.11.2
  • Chart version: 6.42.0
  • Loki version: 3.5.5
  • storage schema v13

We’ve got structured_metadata enabled in helm values and schema v13:
loki:
…snip
limits_config:
allow_structured_metadata: true
volume_enabled: true
schemaConfig:
configs:
- from: "2024-04-01"
index:
period: 24h
prefix: loki_index_
object_store: s3
schema: v13
store: tsdb
…snip
And it seems to run with that too:

kubectl exec -n loki loki-write-0 -c loki -- grep structured /etc/loki/config/config.yaml
allow_structured_metadata: true

Although there’s also an empty running-config.yaml - not sure if this overwrites:

kubectl exec -n loki loki-write-0 -c loki -- cat /etc/loki/runtime-config/runtime-config.yaml
{}

In alloy, I add a structured_metadata stage but i dont see this field in grafana:
loki.process "system" {
forward_to = [loki.write.default.receiver]
…snip..
stage.structured_metadata {
values = {
hostname = "my-static-hostname-string",
uniquefieldname = "thats numberwang",
}
}

Is there something else I need to do to get structured metadata working? All the logs, labels etc are arriving from alloy, just not with any metadata that i can find. Suggestions very welcome!

2

Don’t see anything obviously missing. Are you seeing any error log from loki writer or alloy?

Also I’d try a manual push via Loki’s API and see what happens.

3

Thanks, I did that too - it returned a 204 and i see the entry but shows nothing in metadata in loki, so it seems its loki side (tried structured_metadata as well as structuredMetadata)

echo ‘{
“streams”: [{
“stream”: {“job”: “direct-v3test”},
“values”: [["’“$(date +%s)000000000”‘", “test structured v3”]],
“structuredMetadata”: [
{“name”: “hostname”, “value”: “test-hostname-v3”},
{“name”: “source”, “value”: “manual-push”}
]
}]
}’ | curl -s -H “Content-Type: application/json; charset=utf-8”
-H “X-Scope-OrgID: 1” --data @- ``http://localhost:33100/loki/api/v1/push

4

Can you get the runtime configuration from the /config API endpoint and share that, please?

Also, can you share a screenshot from Grafana on what your test log looks like?