Hey,
I’m new to Grafana. I’m not sure what I’m doing wrong, but I can’t get log level labels to show up in Loki (Grafana Logs Drilldown). I self-deployed the Grafana, Alloy, and Loki stack to my host server. Everything else seems to be working fine.
// alloy-config.river
local.file_match "client_hub_service_files" {
path_targets = [{__path__ = "some_path/*.log"}]
sync_period = "60s"
}
loki.source.file "client_hub_service_logs" {
targets = local.file_match.client_hub_service_files.targets
forward_to = [loki.process.add_labels_client_hub.receiver]
tail_from_end = false
}
loki.process "add_labels_client_hub" {
stage.json {
expressions = {
"timestamp" = "timestamp",
"message" = "message",
"extracted_level" = "level",
"correlation_id" = "correlation_id",
}
}
// Add the value of "extracted_level" from the extracted map as a "level" label
stage.labels {
values = {
"level" = "extracted_level",
"correlation_id" = "correlation_id",
}
}
stage.static_labels {
values = {
job = "jobname",
service_name = "client_hub_service",
env = "production",
}
}
forward_to = [loki.write.grafana_cloud_loki.receiver]
}
// Lead Service Logs
local.file_match "lead_service_files" {
path_targets = [{__path__ = "<path>/*.log"}]
sync_period = "60s"
}
loki.source.file "lead_service_logs" {
targets = local.file_match.lead_service_files.targets
forward_to = [loki.process.add_labels_lead_service.receiver]
tail_from_end = false
}
loki.process "add_labels_lead_service" {
stage.json {
expressions = {
"timestamp" = "timestamp",
"message" = "message",
"level" = "level",
"correlation_id" = "correlation_id",
}
}
stage.static_labels {
values = {
job = "some_job",
service_name = "lead_service",
env = "production",
}
}
forward_to = [loki.write.grafana_cloud_loki.receiver]
}
// Common Loki Write Endpoint
loki.write "grafana_cloud_loki" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
Using this config I can see on the logs (Loki):
level “info”
level_extracted “info”
Please see attached screenshot for what I see on the logs.
I guess your level is only a metadata, not a label.
I would try to use also different name, because this may a problem for discover_log_levels
# Discover and add log levels during ingestion, if not present already. Levels
# would be added to Structured Metadata with name
# level/LEVEL/Level/Severity/severity/SEVERITY/lvl/LVL/Lvl (case-sensitive) and
# one of the values from 'trace', 'debug', 'info', 'warn', 'error', 'critical',
# 'fatal' (case insensitive).
# CLI flag: -validation.discover-log-levels
[discover_log_levels: <boolean> | default = true]
Please check doc for your version for more details.
I did not understand your answer.
You can see in my config above I extracted the “level” metadata from the log and label it on “stage.labels” as “level”.
Wouldn’t be enough to populate log levels drop down?
Example log:
{"context":{},"correlation_id":"req-1745351171750-3vqwjtyx5","level":"info","message":"Server running on port","service":"client-hub-service","timestamp":"2025-04-22T19:46:11.750+00:00"}
loki:3.2.0
alloy:v1.4.0
What will happen if you remove “stage.labels” from the label config? Loki will generate “level” anyway because its feature discover_log_levels is enabled by default—see the linked document.
So you are conflicting with the default Loki feature. How will it behave? I don’t know. Check the document or simply don’t use level but my mylevel name instead.
Yeah, I already removed it from the “stage.labels”, cleaned all the logs (reindex) and created new logs, but it also did not work. My logs level dropdown won’t populate.
Now I only have a single “message” which is part of my log metadata.
Correct, because loki creates level as metadata, not as a label by default. “dropdown” can be generated only for a label, not for metadata.
Good, so do you have any concrete solution, assuming I’ve already removed ‘label’ from ‘stage.labels’?
In the docs, it’s written:
This means that you don’t need to add labels for things inside the log message, such as:
- log level
- log message
- exception name
However looking on the example there, it seems they are doing exactly so (labeling log level).
My current config file:
loki.process "add_labels_client_hub" {
stage.json {
expressions = {
"timestamp" = "timestamp",
"correlation_id" = "correlation_id",
}
}
stage.labels {
values = {
// removed log level label
"correlation_id" = "correlation_id",
}
}
As I said: don’t use level but my mylevel name instead. You must create mylevel label from level log field and then you can create “dropdown” from mylevel label.
Something like that:
loki.process "add_labels_client_hub" {
stage.json {
expressions = {
"timestamp" = "timestamp",
"correlation_id" = "correlation_id",
"level" = "level",
}
}
stage.labels {
values = {
// removed log level label
"correlation_id" = "correlation_id",
"my_level" = "level"
}
}
What do you mean by:
then you can create “dropdown” from mylevel label
Is there any action require from my side, or it should populate by itself after applying the config you suggested above?
