How to add labels with loki.source.syslog

Hello!! I would first like to apologize for using the wrong terminology in my questions. I am new to Grafana Alloy and Loki! I need to monitor syslogs accross my network. I am currently using loki.source.syslog to collect syslogs and write them to loki. How can I apply custom labels based on the ip_address sending the syslog?

I have used discovery.http to send targets and custom labels to prometheus.scrape, but how do I do that with loki? There are no targets and everything references maps.

I know that I can use loki.relabel to change or update labels, but can I use that to add custom labels based on the ip_address in the syslog? It seems like this is to manipulate existing labels in the log.

I am not trying to add dozens of labels, just an application name, location, and environment.

Although it’s not documented, from the source code it looks like a label is added with the remote IP (see alloy/internal/component/loki/source/syslog/internal/syslogtarget/transport.go at b957de68821926812b261df81ac80fcbeadddb05 · grafana/alloy · GitHub)

So you should be able to add a label like this:

loki.process "{NAME}" {
  forward_to = ["{...}"]

  stage.static_labels {
    values = {
      remote_ip = "__syslog_connection_ip_address",
    }
  }
}

Thank you Tony!!!

To confirm, stage.static_labels will add a label called remote_ip to the log and that can be used to search in loki, correct?

Is there a way to take that value and add labels for environment, application, and location? For example, I have the following json,
[
{
“remote_ip”: “1.2.3.4.5”,
“location”: “chicago”,
“environment”: “development”,
“application”: “grafana”
}
]
I want to add the other labels to the log based on the remote_ip.

I looked at loki.relabel and it looks like you can drop, keep, and modify, but not add new labels. Would I use stage.label?

Thank you again for your help!!

Apologies, my previous comment was actually incorrect. You want to use stage.labels to assign value to a label from another label, and use stage.static_labels to hardcode value to a label. So you’d want:

loki.process "{NAME}" {
  forward_to = ["{...}"]

  stage.labels {
    values = {
      "remote_ip" = "__syslog_connection_ip_address", # this assign value of __syslog_connection_ip_address label to remote_ip
    }
  }

  stage.static_labels {
    values = {
      "location" = "chicago",
      "environment" = "development",
      "application" = "something",
    }
  }
}

Could I use something like this?

loki.process “add_labels” {
forward_to = [“{…}”]

stage.labels {
values = {
“remote_ip” = “__syslog_connection_ip_address”,
}
stage.match {
selector = “{remote_ip="1.2.3.4.5"}”
stage.static_labels {
values = {
“location” = “chicago”,
“environment” = “development”,
“application” = “app1”,
}
}
stage.match {
selector = “{remote_ip="9.2.3.4.5"}”
stage.static_labels {
values = {
“location” = “chicago”,
“environment” = “production”,
“application” = “app1”,
}
}
stage.match {
selector = “{remote_ip="4.5.6.7.8"}”
stage.static_labels {
values = {
“location” = “indianapolis”,
“environment” = “development”,
“application” = “app2”,
}
}
}

I also have same use case can you help me.
So i have Linux and Windows Server, I want to add Below Static Values/Labels

  1. Location
  2. IP Address
  3. Rack

How can i achieve them and added them to loki and prometheus?
On Fleet Overview Screen I want to Display Location and IP Address

Are you using syslog as well? syslog has a list of labels that it tries to produce internally, which should cover ip address, but for other external information such as rack you’d need to supply those yourself.

I noob in this part.
Please guide me how can i use syslog.
For Rack and Location in Agent I had added global labels
Please guide me the same how can I add Lables for that instance. They are static value. I can manually input it it’s okay.

See loki.process | Grafana Alloy documentation for static labels.

and what about syslog ?
How to use that ?

can you help me how can i enable syslog ?

prometheus.exporter.self "integrations_alloy" { }

discovery.relabel "integrations_alloy" {
  targets = prometheus.exporter.self.integrations_alloy.targets

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }

  rule {
    target_label = "alloy_hostname"
    replacement  = constants.hostname
  }

  rule {
    target_label = "job"
    replacement  = "integrations/alloy-check"
  }
}

prometheus.scrape "integrations_alloy" {
  targets    = discovery.relabel.integrations_alloy.output
  forward_to = [prometheus.relabel.integrations_alloy.receiver]  

  scrape_interval = "60s"
}

prometheus.relabel "integrations_alloy" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "(prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|alloy_build.*|prometheus_remote_write_wal_samples_appended_total|process_start_time_seconds)"
    action        = "keep"
  }
}

prometheus.remote_write "metrics_service" {
  endpoint {
    url = "{METRICS_URL}"
  }
}

loki.write "grafana_loki" {
  endpoint {
    url = "{LOGS_URL}"
  }
}

discovery.relabel "integrations_node_exporter" {
  targets = prometheus.exporter.unix.integrations_node_exporter.targets

  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }

  rule {
    target_label = "job"
    replacement = "integrations/node_exporter"
  }
}

prometheus.exporter.unix "integrations_node_exporter" {
  disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]

  filesystem {
    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
    mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
    mount_timeout        = "5s"
  }

  netclass {
    ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }

  netdev {
    device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
  }
}

prometheus.scrape "integrations_node_exporter" {
  targets    = discovery.relabel.integrations_node_exporter.output
  forward_to = [prometheus.relabel.integrations_node_exporter.receiver]
}

prometheus.relabel "integrations_node_exporter" {
  forward_to = [prometheus.remote_write.metrics_service.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "node_scrape_collector_.+"
    action        = "drop"
  }
}

loki.source.journal "logs_integrations_integrations_node_exporter_journal_scrape" {
  max_age       = "24h0m0s"
  relabel_rules = discovery.relabel.logs_integrations_integrations_node_exporter_journal_scrape.rules
  forward_to    = [loki.write.grafana_loki.receiver]
}

local.file_match "logs_integrations_integrations_node_exporter_direct_scrape" {
  path_targets = [{
    __address__ = "localhost",
    __path__    = "/var/log/{syslog,messages,*.log}",
    instance    = constants.hostname,
    job         = "integrations/node_exporter",
  }]
}

discovery.relabel "logs_integrations_integrations_node_exporter_journal_scrape" {
  targets = []

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }

  rule {
    source_labels = ["__journal__boot_id"]
    target_label  = "boot_id"
  }

  rule {
    source_labels = ["__journal__transport"]
    target_label  = "transport"
  }

  rule {
    source_labels = ["__journal_priority_keyword"]
    target_label  = "level"
  }
}

loki.source.file "logs_integrations_integrations_node_exporter_direct_scrape" {
  targets    = local.file_match.logs_integrations_integrations_node_exporter_direct_scrape.targets
  forward_to = [loki.write.grafana_loki.receiver]
}

loki.source.syslog "LABEL" {
  listener {
    address = "LISTEN_ADDRESS"
  }
  ...

  forward_to = RECEIVER_LIST
}

I have seen this but i am not able to integrate can you please help

There are some examples here: loki.source.syslog | Grafana Alloy documentation

Hi, I tried doing this to get the connection IP address but it does not work. The value “remote_ip” ended up being populated by the string “__syslog_connection_ip_address” instead of the IP address.
image

PS: I am new to Grafana Loki. So may not be familiar with the various syntax.

Mind sharing your latest config?

I followed the example above.

loki.process "syslog" {
  forward_to = [loki.write.default.receiver]
  stage.static_labels {
    values = {
      remote_ip = "__syslog_connection_ip_address",
    }
  }
}

static_labels sets labels literally, you want to use the labels stage.

2 Likes

oh ok. I guess I must have cut and pasted the wrong section.

anyway I went ahead to use the labels stage. and now the remote_ip label does not appear in the drop-down list at all…


Can you please post a sample syslog (obfuscated data) for example my syslog parser looks like this

loki.process "regex" {
    stage.regex {
        expression = `REDACTED`
    }
    
    forward_to = [loki.write.default.receiver]

    stage.labels {
      values = {
        log_type = "",
        date = "",
        trace = "",
        time2 = "",
        msgid = "",
        message = "",
      }
    }
}

as you can see the labels come from the parsed log file.

Actually I am trying to get the name/IP address of the device that sends the syslogs. I have 2 firewalls sending syslogs to Loki. Currently, I am using 2 different ports to receive the logs. I am hoping that the Loki syslog component is able to extract at least some info of the source device.

From the example by @tonyswumac , there seem to be a variable __syslog_connection_ip_address , but it does seem to get populated for me.

Here are 2 samples of the syslog from my Fortigate Firewall. Maybe one way is to manually use regex to extract the Firewall name but I am hoping there is an easier way that is already built-in.


2024-11-01 19:13:31.752	
<189>1 2024-11-01T11:13:31Z CMC-Fortigate-60F - - - - eventtime=1730459611090721930 tz="+0800" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" srcip=169.254.132.11 srcport=23825 srcintf="root" srcintfrole="undefined" dstip=173.243.140.16 dstport=8888 dstintf="wan1" dstintfrole="wan" srccountry="Reserved" dstcountry="United States" sessionid=5457498 proto=17 action="accept" policyid=0 service="udp/8888" trandisp="noop" app="udp/8888" duration=180 sentbyte=92 rcvdbyte=0 sentpkt=1 rcvdpkt=0 masterdstmac="b4:0c:25:f2:c0:11" dstmac="b4:0c:25:f2:c0:11" dstserver=0
2024-11-01 19:13:10.442	
<189>1 2024-11-01T11:13:10Z CMC-Fortigate-60F - - - - eventtime=1730459590209717610 tz="+0800" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" srcip=192.168.0.50 srcport=50817 srcintf="internal" srcintfrole="lan" dstip=192.168.0.255 dstport=137 dstintf="unknown-0" dstintfrole="undefined" replysrcintf="root" srccountry="Reserved" dstcountry="Reserved" sessionid=5457882 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/137" trandisp="noop" app="netbios forward" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 msg="Connection Failed" mastersrcmac="00:11:32:94:0a:ad" srcmac="00:11:32:94:0a:ad" srcserver=0

Look at this thread

There are also other options such as pattern