Hi everyone,
I use Loki 3.0 and ingest logs in OTLP format with the OpenTelemetry Collector and the otlphttp exporter.
Everything works as expected, but I cannot add the resource attribute host.name as an additional index label. The attribute is instead added as structured metadata.
I followed the instructions here: Ingesting logs to Loki using OpenTelemetry Collector | Grafana Loki documentation
Other resource attributes from the list of default labels like deployment.environment are converted to index labels as expected.
This is the section from Loki’s configuration file:
limits_config:
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
attributes:
- host.name
In the OpenTelemetry Collector debug log, the log record that is sent to Loki looks like this:
2024-05-07T15:48:48.551+0200 info ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
-> service.namespace: Str(demo-namespace)
-> service.name: Str(demo-service)
-> host.name: Str(example.com)
-> deployment.environment: Str(test)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2024-05-07 13:48:47.895773234 +0000 UTC
Timestamp: 2024-05-07 13:48:47.701 +0000 UTC
SeverityText: DEBUG
SeverityNumber: Debug(5)
Body: Str(log content)
Attributes:
-> log.file.name: Str(log.ndjson)
-> log.file.path: Str(/path/to/logs/log.ndjson)
Trace ID: dac9c66b486632ce141cbe9a98712c41
Span ID: 4b9cf39a00f42ce9
Flags: 1
What could I do to store host.name as an index label instead of structured metadata?
Thank you for your help!
Stephan
I’ve not tried using otlp collector with Loki, but according to documentation there is a list of fixed attributes that get mapped to labels. See Ingesting logs to Loki using OpenTelemetry Collector | Grafana Loki documentation
Thanks for you response!
I followed the documentation, there is a section about how to extend the built-in list of attributes:
But these settings do not seem to have any effect in my installation. I also tried to disable the default attributes as described here, but it also did not work:
I’m having the same problem. As a workaround I just changed the default list of resource attributes that gets mapped via the distributor config, but that’s not ideal. Example:
loki:
distributor:
otlp_config:
default_resource_attributes_as_index_labels:
- foo
- bar
There was a bug that was fixed in PR: fix: fix parsing of default per tenant otlp config by sandeepsukhani · Pull Request #12836 · grafana/loki · GitHub. The fix has been released with 3.1.1. Please let us know if you still an issue after the upgrade.
Hi!
I tried adding the otlp_config inside the limits_config on loki version 3.1.1, but it is still not working.
All fields stay as resource attributes.
For the above example, should this work?
limits_config:
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
attributes:
- host.name
2 Likes
Seeing the same issue on 3.2.0
Having the same issue, even in v3.1.1.
I was able to get this to work after some finagling with the structure. Here is my working structure some items are promoted to Resource attributes and others to log attributes
Was there a better solution than this? Since Grafana Alloy and OTELCollector both support configuraing these labels, I’m not really sure what the right config within loki is now anymore. What i personally want is to disable any logic in Loki and let Alloy handle the decision of labels 
but it just does not work
The list of promoted resource attributes is configurable using Loki’s distributor config parameter default_resource_attributes_as_index_labels when using self-managed Loki:
otlp_config:
# List of default otlp resource attributes to be picked as index labels
# CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels
[default_resource_attributes_as_index_labels: <list of strings> | default = [service.name service.namespace service.instance.id deployment.environment cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.pod.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name]]
Problem Description:
I have below OTEL log detail. And I want to add custom index and structured metadata in loki.
I am referencing https://grafana.com/docs/loki/latest/send-data/otel/
I have tried with this and work with resource attribute in loki config setting, all the resource attributes are deleted except service.name and env
otlp_config:
resource_attributes:
attributes_config:
- action: index_label
regex: service.name
- action: index_label
regex: env
- action: drop
regex: .*
The problem arises when I want to remove ObservedTimestamp or any other attribute that isare not belongs to resource attribute (e.g ObservedTimestamp). I have tried below code in loki and this is not working.
log_attributes:
- action: drop
regex: .*
My OTEL log
2024-11-29 14:53:20 2024-11-29T06:53:20.945Z info Logs {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2024-11-29 14:53:20 2024-11-29T06:53:20.945Z info ResourceLog #0
2024-11-29 14:53:20 Resource SchemaURL: https://opentelemetry.io/schemas/1.24.0
2024-11-29 14:53:20 Resource attributes:
2024-11-29 14:53:20 -> env: Str(default)
2024-11-29 14:53:20 -> host.arch: Str(aarch64)
2024-11-29 14:53:20 -> os.type: Str(darwin)
2024-11-29 14:53:20 -> process.pid: Int(79761)
2024-11-29 14:53:20 -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.7+7-LTS)
2024-11-29 14:53:20 -> process.runtime.name: Str(OpenJDK Runtime Environment)
2024-11-29 14:53:20 -> process.runtime.version: Str(17.0.7+7-LTS)
2024-11-29 14:53:20 -> service.instance.id: Str(79ac612c-83db-4374-868f-e9f0568c8125)
2024-11-29 14:53:20 -> service.name: Str(spring-boot-core)
2024-11-29 14:53:20 -> telemetry.distro.name: Str(opentelemetry-spring-boot-starter)
2024-11-29 14:53:20 -> telemetry.distro.version: Str(2.10.0)
2024-11-29 14:53:20 -> telemetry.sdk.language: Str(java)
2024-11-29 14:53:20 -> telemetry.sdk.name: Str(opentelemetry)
2024-11-29 14:53:20 -> telemetry.sdk.version: Str(1.43.0)
2024-11-29 14:53:20 ScopeLogs #0
2024-11-29 14:53:20 ScopeLogs SchemaURL:
2024-11-29 14:53:20 InstrumentationScope com.example.core.filter.HighestPriorityHttpRequestOnceFilter
2024-11-29 14:53:20 LogRecord #0
2024-11-29 14:53:20 ObservedTimestamp: 2024-11-29 06:53:20.549932 +0000 UTC
2024-11-29 14:53:20 Timestamp: 2024-11-29 06:53:20.549498 +0000 UTC
2024-11-29 14:53:20 SeverityText: INFO
2024-11-29 14:53:20 SeverityNumber: Info(9)
2024-11-29 14:53:20 Body: Str(Hello World)
2024-11-29 14:53:20 Trace ID: d528fbe2dec5f1fae0167372f7f9c346
2024-11-29 14:53:20 Span ID: 956361158c3b8cbd
2024-11-29 14:53:20 Flags: 1
2024-11-29 14:53:20 {"kind": "exporter", "data_type": "logs", "name": "debug"}
Does someone know how to delete unwanted attribute that does not belong to resource attribute?
ObservedTimestamp is not attribute, but field of the otlp protocol. Be familiar with OTLP specification. TLDR: you can work only with resource attributes/signal attribute, but not with mandatory protocol fields.
Hello, thanks for your response.
The fields are inserted into loki as structured metadata in loki as shown in below screenshot. Do you mean that we don’t have any ways to remove non resource/signal attribute (structured metadata) for loki currently?
Thanks!
I didn’t say that. But you can’t do that on the OTEL collector side.
BTW: you are hijacking thread with different problem. Your best option is new topic and ask: how to configure Loki, so OTEL fields won’t be indexed as metadata. - of course use google, read the Loki doc, … before posting
