OTLP ingestion from OTEL collector to Loki seems to corrupt my data

I am sending logs to my self-hosted Loki 3.2.x via the OTEL-collector Helm chart using the otel/opentelemetry-collector-contrib image with this config.

config:
  exporters:
    debug:
      verbosity: basic
    otlphttp:
      endpoint: http://xxx.yyy
      tls:
        insecure: true
    otlphttp/loki:
      endpoint: http://loki.xxx.yyy/otlp
      tls:
        insecure: true
  receivers:
    jaeger: null
    zipkin: null
  service:
    extensions:
      - health_check
    pipelines:
      logs:
        receivers:
          - otlp
        exporters:
          - otlphttp/loki
          - debug
        processors:
        - memory_limiter
        - batch
        - resource

      traces:
        receivers:
          - otlp
        exporters:
          - otlphttp
          #- debug
        processors:
        - memory_limiter
        - batch
        - resource
      metrics:
        receivers:
          - otlp
        exporters:
          #- debug
        processors:
          - memory_limiter
          - batch
          - resource

  processors:
    resource:
      attributes:
      - key: k8s.cluster.name
        value: ${environment_short}-aks
        action: upsert

The data seems to arrive to Loki without any obvious issues, but reading it out from my self-hosted Grafana results in this error for the majority of the data:

{
    "results": {
        "A": {
            "error": "failed to parse series labels to categorize labels: 1:2: parse error: unexpected \"=\" in label set, expected identifier or \"}\"",
            "errorSource": "downstream",
            "status": 500
        }
    }
}

For some reason, certain logs work as expected, while the majority of my log entries seem to corrupt the data/labels. Not quite sure what is going on.

Sample data from a successful Grafana query towards data ingested via OTLP:

Common labels: {"ConnectionId":"0HN9MSQJGRG0I","detected_level":"debug","k8s_cluster_name":"d-aks","k8s_container_name":"testteam1-web","k8s_deployment_name":"testteam1-web","k8s_namespace_name":"testteam1","k8s_node_name":"aks-workerpool2-39622677-vmss000000","k8s_pod_name":"testteam1-web-6445867b4-6hf6p","k8s_replicaset_name":"testteam1-web-6445867b4","service_instance_id":"testteam1.testteam1-web-6445867b4-6hf6p.testteam1-web","service_name":"testteam1-web","service_version":"17-01-2025.981","severity_number":"5","severity_text":"Debug","telemetry_sdk_language":"dotnet","telemetry_sdk_name":"opentelemetry","telemetry_sdk_version":"1.11.0"}
Line limit: "1000 (4 displayed)"
Total bytes processed: "1.28  kB"


2025-01-17 12:05:03.720	Connection id "0HN9MSQJGRG0I" stopped.
2025-01-17 12:05:03.718	Connection id "0HN9MSQJGRG0I" disconnecting.
2025-01-17 12:05:03.715	Connection id "0HN9MSQJGRG0I" sending FIN because: "The Socket transport's send loop completed gracefully."
2025-01-17 12:05:03.714	Connection id "0HN9MSQJGRG0I" received FIN.

^ Just to show that ingestion and querying works just fine for some of the log lines

When running a query towards “corrupted” data the query first displays like this:

Then it is stuck loading for a while before it ends up like this:

I have been successfully using Promtail for Loki ingestion for years without similar issues. I am not sure how to go on about debugging or what might be the issue here. The open-telemetry collector does not show any errors or warnings in the debug log output.

Enable debug exporter in your log pipeline and check resource attributes and log line attributes.

Detailed debug output sample from the collector of a part which causes the error when querying in Grafana:


2025-01-17 13:07:33.485		{"kind": "exporter", "data_type": "logs", "name": "debug"}
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(The request path {Path} does not match the path filter)
2025-01-17 13:07:33.485	     -> Path: Str()
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(The request path  does not match the path filter)
2025-01-17 13:07:33.485	SeverityNumber: Debug(5)
2025-01-17 13:07:33.485	SeverityText: Debug
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.7392168 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.7392168 +0000 UTC
2025-01-17 13:07:33.485	LogRecord #0
2025-01-17 13:07:33.485	InstrumentationScope Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware 
2025-01-17 13:07:33.485	ScopeLogs SchemaURL: 
2025-01-17 13:07:33.485	ScopeLogs #4
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(Request did not match any endpoints)
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(Request did not match any endpoints)
2025-01-17 13:07:33.485	SeverityNumber: Debug(5)
2025-01-17 13:07:33.485	SeverityText: Debug
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.7390758 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.7390758 +0000 UTC
2025-01-17 13:07:33.485	LogRecord #0
2025-01-17 13:07:33.485	InstrumentationScope Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware 
2025-01-17 13:07:33.485	ScopeLogs SchemaURL: 
2025-01-17 13:07:33.485	ScopeLogs #3
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(No candidates found for the request path '{Path}')
2025-01-17 13:07:33.485	     -> Path: Str(/)
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(No candidates found for the request path '/')
2025-01-17 13:07:33.485	SeverityNumber: Debug(5)
2025-01-17 13:07:33.485	SeverityText: Debug
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.7390544 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.7390544 +0000 UTC
2025-01-17 13:07:33.485	LogRecord #0
2025-01-17 13:07:33.485	InstrumentationScope Microsoft.AspNetCore.Routing.Matching.DfaMatcher 
2025-01-17 13:07:33.485	ScopeLogs SchemaURL: 
2025-01-17 13:07:33.485	ScopeLogs #2
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(Request reached the end of the middleware pipeline without being handled by application code. Request path: {Method} {Scheme}://{Host}{PathBase}{Path}, Response status code: {StatusCode})
2025-01-17 13:07:33.485	     -> StatusCode: Int(404)
2025-01-17 13:07:33.485	     -> Path: Str(/)
2025-01-17 13:07:33.485	     -> PathBase: Str()
2025-01-17 13:07:33.485	     -> Host: Str(testteam123.d.aks.dummysite.com)
2025-01-17 13:07:33.485	     -> Scheme: Str(http)
2025-01-17 13:07:33.485	     -> Method: Str(GET)
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(Request reached the end of the middleware pipeline without being handled by application code. Request path: GET http://testteam123.d.aks.dummysite.com/, Response status code: 404)
2025-01-17 13:07:33.485	SeverityNumber: Info(9)
2025-01-17 13:07:33.485	SeverityText: Information
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.7395529 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.7395529 +0000 UTC
2025-01-17 13:07:33.485	LogRecord #2
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(Request finished {Protocol} {Method} {Scheme}://{Host}{PathBase}{Path}{QueryString} - {StatusCode} {ContentLength} {ContentType} {ElapsedMilliseconds}ms)
2025-01-17 13:07:33.485	     -> QueryString: Str()
2025-01-17 13:07:33.485	     -> Path: Str(/)
2025-01-17 13:07:33.485	     -> PathBase: Str()
2025-01-17 13:07:33.485	     -> Host: Str(testteam123.d.aks.dummysite.com)
2025-01-17 13:07:33.485	     -> Scheme: Str(http)
2025-01-17 13:07:33.485	     -> Method: Str(GET)
2025-01-17 13:07:33.485	     -> Protocol: Str(HTTP/1.1)
2025-01-17 13:07:33.485	     -> ContentLength: Int(0)
2025-01-17 13:07:33.485	     -> : Empty()
2025-01-17 13:07:33.485	     -> StatusCode: Int(404)
2025-01-17 13:07:33.485	     -> ElapsedMilliseconds: Double(0.6366)
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(Request finished HTTP/1.1 GET http://testteam123.d.aks.dummysite.com/ - 404 0 - 0.6366ms)
2025-01-17 13:07:33.485	SeverityNumber: Info(9)
2025-01-17 13:07:33.485	SeverityText: Information
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.739474 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.739474 +0000 UTC
2025-01-17 13:07:33.485	LogRecord #1
2025-01-17 13:07:33.485	Flags: 1
2025-01-17 13:07:33.485	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.485	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.485	     -> RequestPath: Str(/)
2025-01-17 13:07:33.485	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.485	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.485	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.485	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.485	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.485	     -> {OriginalFormat}: Str(Request starting {Protocol} {Method} {Scheme}://{Host}{PathBase}{Path}{QueryString} - {ContentType} {ContentLength})
2025-01-17 13:07:33.485	     -> QueryString: Str()
2025-01-17 13:07:33.485	     -> Path: Str(/)
2025-01-17 13:07:33.485	     -> PathBase: Str()
2025-01-17 13:07:33.485	     -> Host: Str(testteam123.d.aks.dummysite.com)
2025-01-17 13:07:33.485	     -> Scheme: Str(http)
2025-01-17 13:07:33.485	     -> : Empty()
2025-01-17 13:07:33.485	     -> : Empty()
2025-01-17 13:07:33.485	     -> Method: Str(GET)
2025-01-17 13:07:33.485	     -> Protocol: Str(HTTP/1.1)
2025-01-17 13:07:33.485	Attributes:
2025-01-17 13:07:33.485	Body: Str(Request starting HTTP/1.1 GET http://testteam123.d.aks.dummysite.com/ - - -)
2025-01-17 13:07:33.485	SeverityNumber: Info(9)
2025-01-17 13:07:33.485	SeverityText: Information
2025-01-17 13:07:33.485	Timestamp: 2025-01-17 12:07:28.7390022 +0000 UTC
2025-01-17 13:07:33.485	ObservedTimestamp: 2025-01-17 12:07:28.7390022 +0000 UTC
2025-01-17 13:07:33.484	LogRecord #0
2025-01-17 13:07:33.484	InstrumentationScope Microsoft.AspNetCore.Hosting.Diagnostics 
2025-01-17 13:07:33.484	ScopeLogs SchemaURL: 
2025-01-17 13:07:33.484	ScopeLogs #1
2025-01-17 13:07:33.484	Flags: 1
2025-01-17 13:07:33.484	Span ID: 945f812b6ade3461
2025-01-17 13:07:33.484	Trace ID: e1ea4a4f662dca8ec126f7d5d5172b94
2025-01-17 13:07:33.484	     -> RequestPath: Str(/)
2025-01-17 13:07:33.484	     -> RequestId: Str(0HN9MTGPVHB54:00000001)
2025-01-17 13:07:33.484	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.484	     -> ParentId: Str(0000000000000000)
2025-01-17 13:07:33.484	     -> TraceId: Str(e1ea4a4f662dca8ec126f7d5d5172b94)
2025-01-17 13:07:33.484	     -> SpanId: Str(945f812b6ade3461)
2025-01-17 13:07:33.484	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" completed keep alive response.)
2025-01-17 13:07:33.484	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.484	Attributes:
2025-01-17 13:07:33.484	Body: Str(Connection id "0HN9MTGPVHB54" completed keep alive response.)
2025-01-17 13:07:33.484	SeverityNumber: Debug(5)
2025-01-17 13:07:33.484	SeverityText: Debug
2025-01-17 13:07:33.484	Timestamp: 2025-01-17 12:07:28.7394148 +0000 UTC
2025-01-17 13:07:33.484	ObservedTimestamp: 2025-01-17 12:07:28.7394148 +0000 UTC
2025-01-17 13:07:33.484	LogRecord #2
2025-01-17 13:07:33.484	Flags: 0
2025-01-17 13:07:33.484	Span ID: 
2025-01-17 13:07:33.484	Trace ID: 
2025-01-17 13:07:33.484	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" started.)
2025-01-17 13:07:33.484	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.484	Attributes:
2025-01-17 13:07:33.484	Body: Str(Connection id "0HN9MTGPVHB54" started.)
2025-01-17 13:07:33.484	SeverityNumber: Debug(5)
2025-01-17 13:07:33.484	SeverityText: Debug
2025-01-17 13:07:33.484	Timestamp: 2025-01-17 12:07:28.7386444 +0000 UTC
2025-01-17 13:07:33.484	ObservedTimestamp: 2025-01-17 12:07:28.7386444 +0000 UTC
2025-01-17 13:07:33.484	LogRecord #1
2025-01-17 13:07:33.484	Flags: 0
2025-01-17 13:07:33.484	Span ID: 
2025-01-17 13:07:33.484	Trace ID: 
2025-01-17 13:07:33.484	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" accepted.)
2025-01-17 13:07:33.484	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:07:33.484	Attributes:
2025-01-17 13:07:33.484	Body: Str(Connection id "0HN9MTGPVHB54" accepted.)
2025-01-17 13:07:33.484	SeverityNumber: Debug(5)
2025-01-17 13:07:33.484	SeverityText: Debug
2025-01-17 13:07:33.484	Timestamp: 2025-01-17 12:07:28.7385581 +0000 UTC
2025-01-17 13:07:33.484	ObservedTimestamp: 2025-01-17 12:07:28.7385581 +0000 UTC
2025-01-17 13:07:33.484	LogRecord #0
2025-01-17 13:07:33.484	InstrumentationScope Microsoft.AspNetCore.Server.Kestrel.Connections 
2025-01-17 13:07:33.484	ScopeLogs SchemaURL: 
2025-01-17 13:07:33.484	ScopeLogs #0
2025-01-17 13:07:33.484	     -> k8s.cluster.name: Str(d-aks)
2025-01-17 13:07:33.484	     -> telemetry.sdk.version: Str(1.11.0)
2025-01-17 13:07:33.484	     -> telemetry.sdk.language: Str(dotnet)
2025-01-17 13:07:33.484	     -> telemetry.sdk.name: Str(opentelemetry)
2025-01-17 13:07:33.484	     -> service.version: Str(17-01-2025.982)
2025-01-17 13:07:33.484	     -> service.instance.id: Str(testteam1.testteam1-web-589756bdf5-d8f7m.testteam1-web)
2025-01-17 13:07:33.484	     -> k8s.replicaset.name: Str(testteam1-web-589756bdf5)
2025-01-17 13:07:33.484	     -> k8s.pod.name: Str(testteam1-web-589756bdf5-d8f7m)
2025-01-17 13:07:33.484	     -> k8s.node.name: Str(aks-workerpool2-39622677-vmss000000)
2025-01-17 13:07:33.484	     -> k8s.namespace.name: Str(testteam1)
2025-01-17 13:07:33.484	     -> k8s.deployment.name: Str(testteam1-web)
2025-01-17 13:07:33.484	     -> k8s.container.name: Str(testteam1-web)
2025-01-17 13:07:33.484	     -> service.name: Str(testteam1-web)
2025-01-17 13:07:33.484	Resource attributes:
2025-01-17 13:07:33.484	Resource SchemaURL: 
2025-01-17 13:07:33.484	2025-01-17T12:07:33.484Z	info	ResourceLog #0
2025-01-17 13:07:33.482	2025-01-17T12:07:33.482Z	info	Logs	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 9}

Sample of part which works fine when querying in Grafana:

2025-01-17 13:08:33.573		{"kind": "exporter", "data_type": "logs", "name": "debug"}
2025-01-17 13:08:33.573	Flags: 0
2025-01-17 13:08:33.573	Span ID: 
2025-01-17 13:08:33.573	Trace ID: 
2025-01-17 13:08:33.573	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" stopped.)
2025-01-17 13:08:33.573	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:08:33.573	Attributes:
2025-01-17 13:08:33.573	Body: Str(Connection id "0HN9MTGPVHB54" stopped.)
2025-01-17 13:08:33.573	SeverityNumber: Debug(5)
2025-01-17 13:08:33.573	SeverityText: Debug
2025-01-17 13:08:33.573	Timestamp: 2025-01-17 12:08:28.7383097 +0000 UTC
2025-01-17 13:08:33.573	ObservedTimestamp: 2025-01-17 12:08:28.7383097 +0000 UTC
2025-01-17 13:08:33.573	LogRecord #1
2025-01-17 13:08:33.573	Flags: 0
2025-01-17 13:08:33.573	Span ID: 
2025-01-17 13:08:33.573	Trace ID: 
2025-01-17 13:08:33.573	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:08:33.573	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" disconnecting.)
2025-01-17 13:08:33.573	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:08:33.573	Attributes:
2025-01-17 13:08:33.573	Body: Str(Connection id "0HN9MTGPVHB54" disconnecting.)
2025-01-17 13:08:33.573	SeverityNumber: Debug(5)
2025-01-17 13:08:33.573	SeverityText: Debug
2025-01-17 13:08:33.573	Timestamp: 2025-01-17 12:08:28.738083 +0000 UTC
2025-01-17 13:08:33.573	ObservedTimestamp: 2025-01-17 12:08:28.738083 +0000 UTC
2025-01-17 13:08:33.573	LogRecord #0
2025-01-17 13:08:33.573	InstrumentationScope Microsoft.AspNetCore.Server.Kestrel.Connections 
2025-01-17 13:08:33.573	ScopeLogs SchemaURL: 
2025-01-17 13:08:33.573	ScopeLogs #1
2025-01-17 13:08:33.573	Flags: 0
2025-01-17 13:08:33.573	Span ID: 
2025-01-17 13:08:33.573	Trace ID: 
2025-01-17 13:08:33.573	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" sending FIN because: "{Reason}")
2025-01-17 13:08:33.573	     -> Reason: Str(The Socket transport's send loop completed gracefully.)
2025-01-17 13:08:33.573	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:08:33.573	Attributes:
2025-01-17 13:08:33.573	Body: Str(Connection id "0HN9MTGPVHB54" sending FIN because: "The Socket transport's send loop completed gracefully.")
2025-01-17 13:08:33.573	SeverityNumber: Debug(5)
2025-01-17 13:08:33.573	SeverityText: Debug
2025-01-17 13:08:33.573	Timestamp: 2025-01-17 12:08:28.7387579 +0000 UTC
2025-01-17 13:08:33.573	ObservedTimestamp: 2025-01-17 12:08:28.7387579 +0000 UTC
2025-01-17 13:08:33.573	LogRecord #1
2025-01-17 13:08:33.573	Flags: 0
2025-01-17 13:08:33.573	Span ID: 
2025-01-17 13:08:33.573	Trace ID: 
2025-01-17 13:08:33.573	     -> {OriginalFormat}: Str(Connection id "{ConnectionId}" received FIN.)
2025-01-17 13:08:33.573	     -> ConnectionId: Str(0HN9MTGPVHB54)
2025-01-17 13:08:33.573	Attributes:
2025-01-17 13:08:33.573	Body: Str(Connection id "0HN9MTGPVHB54" received FIN.)
2025-01-17 13:08:33.573	SeverityNumber: Debug(5)
2025-01-17 13:08:33.573	SeverityText: Debug
2025-01-17 13:08:33.573	Timestamp: 2025-01-17 12:08:28.7376904 +0000 UTC
2025-01-17 13:08:33.573	ObservedTimestamp: 2025-01-17 12:08:28.7376904 +0000 UTC
2025-01-17 13:08:33.573	LogRecord #0
2025-01-17 13:08:33.573	InstrumentationScope Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets 
2025-01-17 13:08:33.573	ScopeLogs SchemaURL: 
2025-01-17 13:08:33.573	ScopeLogs #0
2025-01-17 13:08:33.573	     -> k8s.cluster.name: Str(d-aks)
2025-01-17 13:08:33.573	     -> telemetry.sdk.version: Str(1.11.0)
2025-01-17 13:08:33.573	     -> telemetry.sdk.language: Str(dotnet)
2025-01-17 13:08:33.573	     -> telemetry.sdk.name: Str(opentelemetry)
2025-01-17 13:08:33.573	     -> service.version: Str(17-01-2025.982)
2025-01-17 13:08:33.573	     -> service.instance.id: Str(testteam1.testteam1-web-589756bdf5-d8f7m.testteam1-web)
2025-01-17 13:08:33.573	     -> k8s.replicaset.name: Str(testteam1-web-589756bdf5)
2025-01-17 13:08:33.573	     -> k8s.pod.name: Str(testteam1-web-589756bdf5-d8f7m)
2025-01-17 13:08:33.573	     -> k8s.node.name: Str(aks-workerpool2-39622677-vmss000000)
2025-01-17 13:08:33.573	     -> k8s.namespace.name: Str(testteam1)
2025-01-17 13:08:33.573	     -> k8s.deployment.name: Str(testteam1-web)
2025-01-17 13:08:33.573	     -> k8s.container.name: Str(testteam1-web)
2025-01-17 13:08:33.573	     -> service.name: Str(testteam1-web)
2025-01-17 13:08:33.573	Resource attributes:
2025-01-17 13:08:33.573	Resource SchemaURL: 
2025-01-17 13:08:33.573	2025-01-17T12:08:33.573Z	info	ResourceLog #0
2025-01-17 13:08:33.573	2025-01-17T12:08:33.572Z	info	Logs	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 4}




OMG - yours logs must be read from bottom to top; otherwise, they don’t make sense. You should note that.

I guess empty attributes : Empty() may be causing parsing issues on the Grafana side. Delete them on the collector side. I think a transform processor should be able to do that.

Good catch noticing that, it seems that you are correct. I tried the following syntax and it seems to work.

    pipelines:
      logs:
        receivers:
          - otlp
        exporters:
          - otlphttp/loki
          - debug
        processors:
        - memory_limiter
        - attributes/empty
        - batch
        - resource

  processors:
    attributes/empty:
      actions:
        - key: "empty"
          # match empty string
          pattern: ^\s*$
          action: delete
1 Like