Loki detected_level as an Index Label

Grafana Loki
1

Hello all,

I’ve read the article Label best practices and respect the considerations regarding re-evaluating level field to structured metadata instead of index labels.

But anyway we have a requirement to use log_level as an index label for searching.

We use OTEL collector → Loki chain, so what I’ve tried to do:

  1. Use OTEL resource attribute as an index label in Loki - it partially works, but if we use batching with otel collector, some banch of logs with different severity will be marked as one log level. Now it’s obvious for me, because resource attributes is not log level attributes, their above of it.
  2. Tried to use Logs attributes from OTEL as index labels - failed, because loki config limits_config:otlp_config:log_attributes -action supports only structured_metadata (not index_label. like resource_attributes)
    Couple thoughts:
  3. detected_level seems the Loki level entity so I’m ok to switch it to labels on the whole Loki installation, but can’t find any settings for that.
  4. I also ok to use another label (for example custom_log_level) as an index label, but again, I can’t use log attributes as an index label and resource attributes is not fit for that.

Am I missing something? Or It’s not possible to use detected_level / custom_level as a index labels in Loki now?

(Version of Loki: 3.3.2, OTEL Collector - 0.114.0)

2

Try:

processors:
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - set(resource.attributes["log_level"], attributes["log_level"])
3

Hello @jangaraj , thank you for your reply. This is on 99% of what I’ve tried to do, except attributes[“log_level”] - there is no attribute like this, cause nobody created it, but we can do it ourselves to use severity_text. So my config:

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["test.log.level"], severity_text) # add to Log level attribute
          - set(resource.attributes["test.log.level"], attributes["test.log.level"]) # add from log level attributes to Resource attributes

And what I get in OTEL collector logs:

ResourceLog #0
Resource SchemaURL:
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.30.0)
     -> service.name: Str(shoppingcart)
     -> service.instance.id: Str(instance-12)
     -> app.attribute: Str(test.attribute)
     -> k8s.pod.ip: Str(10.189.40.76)
     -> k8s.pod.name: Str(python)
     -> k8s.namespace.name: Str(tests)
     -> k8s.pod.start_time: Str(2025-02-10T13:41:48Z)
     -> k8s.pod.uid: Str(a63f0e0d-6433-473e-8f49-412b6921a4f7)
     -> k8s.node.name: Str(ip-10-189-43-176.eu-west-1.compute.internal)
     -> test.log.level: Str(ERROR)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope myapp.area2
LogRecord #0
ObservedTimestamp: 2025-02-12 11:09:21.963794461 +0000 UTC
Timestamp: 2025-02-12 11:09:21.963725824 +0000 UTC
SeverityText: WARN
SeverityNumber: Warn(13)
Body: Str(Jail zesty vixen who grabbed pay from quack.)
Attributes:
     -> code.filepath: Str(//otel.py)
     -> code.function: Str(<module>)
     -> code.lineno: Int(48)
     -> test.log.level: Str(WARN)
Trace ID:
Span ID:
Flags: 0
LogRecord #1
ObservedTimestamp: 2025-02-12 11:09:21.96392148 +0000 UTC
Timestamp: 2025-02-12 11:09:21.963899648 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(The five boxing wizards jump quickly.)
Attributes:
     -> code.filepath: Str(//otel.py)
     -> code.function: Str(<module>)
     -> code.lineno: Int(49)
     -> test.log.level: Str(ERROR)
Trace ID:
Span ID:
Flags: 0
LogRecord #2
ObservedTimestamp: 2025-02-12 11:09:21.964354608 +0000 UTC
Timestamp: 2025-02-12 11:09:21.964335616 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(Hyderabad, we have a major problem.)
Attributes:
     -> code.filepath: Str(//otel.py)
     -> code.function: Str(<module>)
     -> code.lineno: Int(56)
     -> test.log.level: Str(ERROR)
Trace ID: 2401bf5fab54e6d09fb09e2fb3958570
Span ID: 790905b5e0cc1ab4
Flags: 1
        {"kind": "exporter", "data_type": "logs", "name": "debug"}

So as you see - we have a 3 Log entities in one bunch with different severity (1 with WARN, 2 with ERROR)
and because we put secerity_text to Resource Attribute we add the same log level to all of this bunch (test.log.level: Str(ERROR)) - what is not true.

We also have Log Level attribute (test.log.level again, but not in Resource Attribute section), which is different in each log, but I can’t find a way to use Log Attribute as a Index Label in Loki (only Resource Attribute supports the verb add: index_label)

1 Like
4

Got it. Try:

processors:
  groupbyattrs:
    keys:
      - test.log.level

Of course, then Loki must be configured to index the resource attribute “test.log.level” as a label.

1 Like
5

And it looks really what I need.
So, the total solution looks like:

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["test.log.level"], severity_text) # add to Log attribute, also need for grouping
          - set(resource.attributes["test.log.level"], attributes["test.log.level"]) # add from log level attributes to Resource attributes
  groupbyattrs:
     keys:
     - test.log.level # use Log attribute for grouping, it will split the group with different severity

of course don’t forget to enable processors in your pipeline

Looks like a workaround in general (I mean if we speak about log_level as index label), but it should work.

Thank you @jangaraj , really appreciate for clue regarding groupbyattrs processor

1 Like