Grafana Alloy Relabel Help - Parsing syslog

Grafana Alloy
, ,
1

I have a syslog server taking logs from outside hosts, I’d like to have the log host show up under instances so I’m working to reassign the instance label with the host that’s in syslog:

loki.relabel "syslog_relabels" {
    //extract the second space delimited field from the log line and associate it to instance
    //this is for hosts with other hosts sending rsyslog information to it
    rule {
        action = "replace"
        source_labels = ["__log__"] 
        regex = `^\S+\s+(\S+)`
        target_label = "instance" 
    }

    forward_to = [loki.write.to_loki.receiver]
}

While alloy does start, the relabel seems to do nothing, what am I missing?

2
  1. Can you share your entire Alloy configuration (or the relevant loki parts)?
  2. Please share an example of your logs.
3

Hi @tonyswumac - certainly. Here’s my currently functional config with the items not quite working commented out, at this point I’ve tried using loki.relabel and loki.process, and have moved from trying to update the existing instance label to using a new label definition of hostname (which is working for the journal part of the file, just not the syslogs section):

// /////////////
// LOGGING OPTIONS
// /////////////
logging {
  level  = "warn"    // Options: "debug", "info", "warn", "error"
  format = "logfmt"   // Options: "logfmt", "json"
}

// /////////////
// JOURNAL
// /////////////

// Collect logs from systemd journal for node_exporter integration
loki.source.journal "local_journal" {
  // Only collect logs from the last 24 hours
  max_age       = "24h0m0s"
  // Apply relabeling rules to the logs
  relabel_rules = discovery.relabel.local_journal.rules
  //Define some labels
    labels        = {
      component = "journal_export",
      instance = constants.hostname,
      hostname = constants.hostname,
      job = "journal_export",
    }
  // Send logs to the local Loki instance
  forward_to    = [loki.write.to_loki.receiver]
}

// Define relabeling rules for systemd journal logs
discovery.relabel "local_journal" {
  targets = []

  rule {
    // Extract systemd unit information into a label
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }

  rule {
    // Extract boot ID information into a label
    source_labels = ["__journal__boot_id"]
    target_label  = "boot_id"
  }

  rule {
    // Extract transport information into a label
    source_labels = ["__journal__transport"]
    target_label  = "transport"
  }

  rule {
    // Extract log priority into a level label
    source_labels = ["__journal_priority_keyword"]
    target_label  = "level"
  }

}

// /////////////
// LOGS
// /////////////

// Define which log files to collect for node_exporter
local.file_match "local_syslog" {
  path_targets = [{
    // Target localhost for log collection
    __address__ = "localhost",
    // Collect standard system logs
    //__path__    = "/var/log/{syslog,messages,*.log}",
    __path__    = "/var/log/syslog",
    // Add instance label with hostname
    instance    = constants.hostname,
    // Add job label for logs
    job         = "syslog_export",
  }]
}

// Collect logs from files for node_exporter
loki.source.file "local_syslog" {
  // Use targets defined in local.file_match
  targets    = local.file_match.local_syslog.targets
  // Send logs to the local Loki instance
  forward_to = [loki.write.to_loki.receiver]
}

//loki.process "syslog_relabels" {
    // For the syslog server, grab the hostname and associate it to the instance value
    // Also if omada, <ErrCount>0</ErrCount> is caught, downgrade that to info from error
//    stage.regex {
//        source = "__log__"
//        expression = `^\S+\s+(\S+)`
//    }
//
//    stage.labels {
//        values = {
//            hostname = "$1",
//        }
//    }

//    forward_to = [loki.write.to_loki.receiver]
//}

//loki.relabel "syslog_relabels" {
    //extract the second space delimited field from the log line and associate it to instance
    //this is for hosts with other hosts sending rsyslog information to it
//    rule {
//        action = "replace"
//        source_labels = "__log__" 
//        regex = `^\S+\s+(\S+)`
//        target_label = "hostname" 
//    }
//
//    forward_to = [loki.write.to_loki.receiver]
//}



// /////////////
// OUT TO LOKI
// /////////////

// Define where to send logs for storage
loki.write "to_loki" {
    endpoint {
        // Send logs to a locally running Loki instance
        url ="http://syslog.montysplace.local:3100/loki/api/v1/push"
    }
}

Here’s a syslog excerpt with two different hostnames I’m trying to capture and assign:

2025-07-25T16:50:46.512468-05:00 syslog systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2025-07-25T16:51:25.071534-05:00 omada.montysplace.local  2025-07-25 16:49:28 Home Omada Controller-Home - - - DHCP Server allocated IP address
4

Ok, I’ve figured it out, my issue was not embedding the capturing group label into the regex and using the labels_from_groups arguement. Here’s the working loki.process in case it helps someone else:

loki.process "remote_host_logs_relabels" {
    stage.regex {
        expression = "^[^ ]+ (?P<hostname>[^ ]+)"
        labels_from_groups = true
    }
    forward_to = [loki.write.to_loki.receiver]
}