Log level don't show on the graph

[lengochungptit]

Hi teams,
I want to create a visualization that will count each log level and show it in a graph.
Here is my query:
sum by(severity,level) (count_over_time({source=“sys-varlogs”,zone=“Staging”, vm_name=“lum1-cicd-1-a”} | json | logfmt | drop error , error_details [$__range]))
For the logs that have “severity” and “level” fields. My query is work.

But with the logs that don’t have “severity” and “level” fields. It just only has values of log-level such as “INFO”, “WARN”, “DEBUG”, “DDEBUG”, my query is not work.

As you can see in the image: in the logs sample panel, I can see the log level, but in the graph, I can not see it. All of them as the name: “value”

Is my query is correct?

[tonyswumac]

logfmt is for logs that have a key=value structure. For your second example you’ll want to use pattern. See Log queries | Grafana Loki documentation.

[lengochungptit]

Thanks for your advice @tonyswumac,

I added a new series, it works with some log sources.

But with other sources, there is a conflict between 2 series:

I have many log sources, they have different formats, and can not use 1 query for all of them. Is there any way to add a condition to define which source, use which query?

[tonyswumac]

You should separate your log streams by labels. For example, if you have multiple log files being scraped from a VM and they all have different format, perhaps adding a filename label would be a good idea so you can parse different logs using different logic.

[lengochungptit]

Many thanks @tonyswumac

[lengochungptit]

Hi @tonyswumac
I have Jenkins log format like this:

2024-03-04 10:27:59.115+0000 [id=38118] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work

I’m trying to use pattern to separate log-level from the log,
The query like this:
sum by(severity,level) (count_over_time({source=“jenkins”,zone=“Staging”, vm_name=“lum1-cicd-1-a”} | pattern <date> <time> <level> <_> <message> <_> | drop error , error_details [$__range]))
But I can not separate this string:
“[id=38118] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1:”

How can I separate the log-level from this string?

[tonyswumac]

Try | pattern "<date> <time> [<id>] <level> <_> <message>", see LogQL Analyzer | Grafana Loki documentation.

[lengochungptit]

Thanks @tonyswumac ,
I tried, but still can not separate it:

I think I need to do something to re-format the log in Promtail, but I don’t know how to do.
Could you give me the advice?

[lengochungptit]

thanks @tonyswumac

It’s done. I found a way to add the level labels to the logs using regex. I just added the pipeline_stage to the promtail-config.yaml.