Alloy stage.template not working

Grafana Alloy
1

I’m trying to use stage.template stage to combine multiple fields into output. However whatever i configure template with, nothing works and output is always returned empty. I spent countless hours trying to get the stage working, but fruitlessly. That is very weird because when i leave template out completely, the labels level and output message will get extracted to log lines succesfully.

          stage.match {
            selector = "{app=~\"(am.*)\"} != \"error.stack\""

            stage.json {
              expressions = {
                message = "message",
                level = "level",
              }
            }

            stage.labels {
              values = {
                level = "level",
              }
            }

            stage.template {
              source   = "fmt"
              template = "{{ .level }} {{ .message }}"
            }

            stage.output {
              source = "fmt"
            }
          }

I am using latest Alloy v1.8.3

2

Don’t see anything obviously wrong. Do you have a couple of samples of your logs?

3

Following JSON lines:

{"service":"am-booter","environment":"beta","@timestamp":"2025-04-23T06:08:34.57440483Z","@version":"1.0.0","message":"HikariPool-1 - Shutdown initiated...","logger_name":"com.zaxxer.hikari.HikariDataSource","thread_name":"SpringApplicationShutdownHook","level":"INFO"}

{"service":"am-booter-invest","environment":"alpha","@timestamp":"2025-04-23T06:02:27.367011582Z","@version":"1.0.0","message":"A message has been sent to AMQP exchange=input-exchange.","logger_name":"c.c.a.c.a.amqp.AmqpClient$Companion","thread_name":"http-nio-8080-exec-7","level":"INFO","req.requestURI":"/run","req.xForwardedFor":"103.202.232.124","request.user_id":"22f955@us-west-1_0vFh","req.method":"POST","req.remoteHost":"14.139.00.00","req.requestURL":"https://alpha-booter-EKS-NLB-3buy998bad2.elb.us-east-1.amazonaws.com/run"}

After implementing stage.template, output is: “” for both of them.

4

I suspect your match here is not working:

stage.match {
  selector = "{app=~\"(am.*)\"} != \"error.stack\""

I tested your config and it seems to be working. I omitted the match part because I am not sure what you are matching for.

Test.log:

{"service":"am-booter","environment":"beta","@timestamp":"2025-04-23T06:08:34.57440483Z","@version":"1.0.0","message":"HikariPool-1 - Shutdown initiated...","logger_name":"com.zaxxer.hikari.HikariDataSource","thread_name":"SpringApplicationShutdownHook","level":"INFO"}
{"service":"am-booter-invest","environment":"alpha","@timestamp":"2025-04-23T06:02:27.367011582Z","@version":"1.0.0","message":"A message has been sent to AMQP exchange=input-exchange.","logger_name":"c.c.a.c.a.amqp.AmqpClient$Companion","thread_name":"http-nio-8080-exec-7","level":"INFO","req.requestURI":"/run","req.xForwardedFor":"103.202.232.124","request.user_id":"22f955@us-west-1_0vFh","req.method":"POST","req.remoteHost":"14.139.00.00","req.requestURL":"https://alpha-booter-EKS-NLB-3buy998bad2.elb.us-east-1.amazonaws.com/run"}

config:

loki.process "process_logs" {
  forward_to = [<FORWARD>]

    stage.json {
       expressions = {
         message = "message",
         level = "level",
       }
     }

     stage.labels {
       values = {
       level = "level",
     }
   }

   stage.template {
     source   = "fmt"
     template = "{{ .level }} {{ .message }}"
   }

   stage.output {
     source = "fmt"
   }
}

Perhaps share your entire alloy configuration?

5

Thank you for the reply. I still face same problems with the template stage, when leaving out the matching block entirely, resulting in empty output

        logging {
          level = "error"
          format = "logfmt"
        }

        // LOKI WRITE ENDPOINT
        loki.write "loki" {
          endpoint {
            url = "https://loki.engineering/loki/api/v1/push"
          }
        }

        // LOKI LOGS

        discovery.kubernetes "pod" {
          role = "pod"
        }

        discovery.relabel "pod_logs" {
          targets = discovery.kubernetes.pod.targets

          rule {
            source_labels = ["__meta_kubernetes_namespace"]
            action = "replace"
            target_label = "namespace"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_name"]
            action = "replace"
            target_label = "pod"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_container_name"]
            action = "replace"
            target_label = "container"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
            action = "replace"
            target_label = "app"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
            action = "replace"
            target_label = "__path__"
            separator = "/"
            replacement = "/var/log/pods/*$1/*.log"
          }
        }

        loki.source.kubernetes "pod_logs" {
          targets    = discovery.relabel.pod_logs.output
          forward_to = [loki.process.pod_logs.receiver]
        }

        loki.process "pod_logs" {
          stage.static_labels {
            values = {
              cluster = "platform-engineering",
            }
          }

          stage.json {
            expressions = {
              message = "message",
              level = "level",
            }
          }

          stage.labels {
            values = {
              level = "level",
            }
          }

          stage.template {
            source   = "fmt"
            template = "{{ .level }} {{ .message }}"
          }

          stage.output {
            source = "fmt"
          }

          forward_to = [loki.write.loki.receiver]
        }

Screenshot 2025-05-22 at 12.38.33
Screenshot 2025-05-22 at 12.38.33630×1352 91.8 KB

6

Can you take out stage.output and see what the result looks like, please?

7

After removal, no logs appear at all

8

Then that means you have something somewhere filtering out or dropping all the logs.

Can you share your entire Alloy configuration, please?

9

Configuration is the pretty much the same as before. Am i the only one, who doesn’t get stage.template to work ? Again, if i leave out the template and set output source to message, it’s working as expected:

     logging {
          level = "error"
          format = "logfmt"
        }

        // LOKI WRITE ENDPOINT
        loki.write "loki" {
          endpoint {
            url = "https://loki.engineering/loki/api/v1/push"
          }
        }

        // LOKI LOGS

        discovery.kubernetes "pod" {
          role = "pod"
        }

        discovery.relabel "pod_logs" {
          targets = discovery.kubernetes.pod.targets

          rule {
            source_labels = ["__meta_kubernetes_namespace"]
            action = "replace"
            target_label = "namespace"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_name"]
            action = "replace"
            target_label = "pod"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_container_name"]
            action = "replace"
            target_label = "container"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
            action = "replace"
            target_label = "app"
          }

          rule {
            source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
            action = "replace"
            target_label = "__path__"
            separator = "/"
            replacement = "/var/log/pods/*$1/*.log"
          }
        }

        loki.source.kubernetes "pod_logs" {
          targets    = discovery.relabel.pod_logs.output
          forward_to = [loki.process.pod_logs.receiver]
        }

        loki.process "pod_logs" {
          stage.static_labels {
            values = {
              cluster = "platform-engineering",
            }
          }

          stage.json {
            expressions = {
              message = "message",
            }
          }

          stage.template {
            source   = "fmt"
            template = "message: {{ .message }}"
          }

          stage.output {
            source = "fmt"
          }

          forward_to = [loki.write.loki.receiver]
        }
10

with this configuration log output is always message: