I’m trying to write Opentelemetry logs to Loki pushing them from my dotnet service directly to Loki with Opentelemetry packages, but ran into a problem. When I’m trying to get logs in grafana I get an error
could not write JSON response: 1:2: parse error: unexpected "=" in label set, expected identifier or "}"
After researching I realized that the problem was only with some of the log entries.
Service have a metrics endpoint for prometheus. And when Prometheus collects metrics, the service writes a log entry like this:
I assume that loki use string “application/openmetrics-text;+version=1.0.0;+charset=utf-8” as common label and falling when I’m trying to get logs.
Does loki have some settings to do with this case? This label creates automatically by Opentelemetry package, so I don’t know what to do with it. I can disable Microsoft.AspNetCore.Hosting.Diagnostics logs but I think I need to resolve this issue on loki side or on the Otel package side.
Loki, but itself, doesn’t parse or inject labels. All labels come from your app. I don’t use opentelemetry in .net, so I can’t tell you exactly what to do, but you should be able to configure it accordingly and remove labels that you do not want.
Yes, this is one of the options. I’m not sure I can do this in code without using some middleware like Otel Collector, but of course that’s not a question to loki and I check for this possibility in Otel package.
I’m asking here, because Loki support Otel logs, and this is default behavior of official Openetelemtry packages. I didn’t add any custom labels. So it’s looking strange that loki falls here.
I have the same issue. If I look at the fields in the logs I can see “undefined” field which is weird. I tried to follow a recommendation to remove empty entries via alloy, but it didn’t help.
Here is how I tried to remove them:
otelcol.processor.attributes "remove_empty_attributes" { // Add the empty attribute processor
actions = [
{
key = "*" // Apply to all attributes. Consider restricting this to specific attribute sets (like "attributes" on spans) if needed.
pattern = "^\\s*$"
action = "delete"
},
]
}
IMHO Loki doesn’t have a problem with ‘=’ in label values.
Problem has only a Grafana (at least explore feature), which is trying to do a own “magic” (parsing a labels/metadata).
Does it works in the standard panel query (not explore)?
When you see “undefined” that looks like something has ingested that “undefined” value there. You can use live debugging alloy feature and find how it looks like on the alloy level and drop it there.
I tried to get logs via loki CLI and CLI also gave 500 error. So it’s not a grafana issue.
I also asked about this problem at Stackoverflow. There they suggested me a temporary solution - to delete empty attributes in the Otel collector. This worked for me. Also today in the answers I saw a link to an issue in opentelemetry-dotnet.
So the problem is that loki can’t handle empty attributes correctly. And another problem is that the Otel package sends empty attributes.
OK, I think that’s a problem of data models - edge case for empty labels.
Loki is inspired by Prometheus:
Labels with an empty label value are considered equivalent to labels that do not exist.
But OTEL model:
Attribute values expressing a numerical value of zero, an empty string, or an empty array are considered meaningful and MUST be stored and passed on to processors / exporters.
So it makes sense to delete empty attributes before ingesting OTEL-compatible logs into Loki.
Nope, it fails with the same error. I’ll try empty attributes removal, but it seems like in my case there are no empy attributes, there is an issue with parsing a long log string with different structures in it.
Here is the example of log record which causes the issue:
I’d appreciate if someone could link a doc for rewriting logs in Alloy. My current plan is to remove all special symbols from certain types of records.
