Grafana Alloy Syslog Labeling Issues

Grafana Alloy
, , ,
1

I am having some issues converting a working Grafana Agent (static mode) config to a working Grafana Alloy config.

The following is the working Grafana Agent (static mode) config:

configs:
  - name: base
    scrape_configs:
      - job_name: syslog
        syslog:
          listen_address: localhost:1514
          labels:
            job: syslog
            service: "{{ customer.service }}"
            environment: "{{ deployment.environment }}"
        relabel_configs:
          - source_labels: [__syslog_message_hostname]
            target_label: host
          - source_labels: [__syslog_message_severity]
            target_label: level
          - source_labels: [__syslog_message_app_name]
            target_label: application
          - source_labels: [__syslog_message_facility]
            target_label: facility
          - source_labels: [__syslog_connection_hostname]
            replacement: "{{ansible_hostname}}"
            target_label: connection_hostname

This successfully scrapes the logs from the remote port configures by rsyslog and labels them in Loki.

The following Alloy config (I’ve tried several different version) is not working the same:

loki.source.syslog “remote_syslog” {
listener {
address = “0.0.0.0:1514”
labels = {
component = “loki.source.syslog”,
protocol = “tcp”,
job = “va”,
service = “{{ customer.service }}”,
environment = “{{ deployment.environment }}”,
}
}

listener {
address = “0.0.0.0:1514”
protocol = “udp”
labels = {
component = “loki.source.syslog”,
protocol = “udp”,
job = “va”,
service = “{{ customer.service }}”,
environment = “{{ deployment.environment }}”,
}
}

forward_to = [loki.write.logs_base.receiver]
}

loki.relabel “remote_syslog” {
forward_to = [loki.write.logs_base.receiver]
rule {
source_labels = [“__syslog_message_hostname”]
target_label = “host”
}

    rule {
            source_labels = ["__syslog_message_severity"]
            target_label  = "level"
    }

    rule {
            source_labels = ["__syslog_message_app_name"]
            target_label  = "application"
    }

    rule {
            source_labels = ["__syslog_message_facility"]
            target_label  = "facility"
    }

    rule {
            source_labels = ["__syslog_connection_hostname"]
            replacement   = "{{ ansible_hostname }}"
            target_label  = "connection_hostname"
    }

}

It simply labels the static labels provided in the “listener” block but ignored any other labels.

Any help is appreciated!

2

Try forwarding from your syslog block to loki.relabel.remote_syslog.receiver instead.

3

Hi Tony:

Thanks again for your help. That did help things a little bit … now I’m getting the connection_hostname labeled properly, but it does not seem to discovery and relabel the other syslog labels.

For example, in the static mode config:

            source_labels = ["__syslog_message_app_name"]
            target_label  = "application"

This gets populated with the syslog component from the host system, but when using Alloy this is ignored and no label is found in Loki.

Thanks for your help.

4

Maybe try this, essentially setting relabel rules in the rsyslog listener instead, and see if it works:

loki.source.syslog "remote_syslog" {
  forward_to    = [loki.write.logs_base.receiver]
  relabel_rules = loki.relabel.remote_syslog.rules

  listener {
    address = "0.0.0.0:1514"
    labels = {
      component = "loki.source.syslog",
      protocol = "tcp",
      job = "va",
      service = "{{ customer.service }}",
      environment = "{{ deployment.environment }}",
    }
  }

  listener {
    address = "0.0.0.0:1514"
    protocol = "udp"
    labels = {
      component = "loki.source.syslog",
      protocol = "udp",
      job = "va",
      service = "{{ customer.service }}",
      environment = "{{ deployment.environment }}",
    }
  }
}

loki.relabel "remote_syslog" {
  forward_to = [loki.write.logs_base.receiver]

  rule {...}
}

If that still doesn’t work, do some tcpdump just to make sure the rsyslog labels are sent correctly.

5

Hello again,

I did run a tcpdump and the labels are correct. When I use static mode it relabels them perfectly, but when I use Alloy, it only adds the static labels, and seemingly ignores the relabeling block. Here is the updates config with my rsyslog config included:

Logs are coming in on UDP port 514 →
Rsyslog then sends them to port 1514->

ruleset(name=“remote”){
action(type=“omfwd”
Target=“localhost”
Port=“1514”
Protocol=“tcp”
Template=“RSYSLOG_SyslogProtocol23Format”
TCP_Framing=“octet-counted”
KeepAlive=“on”
action.resumeRetryCount=“-1”
queue.type=“linkedlist”
queue.size=“50000”
)
}

module(load=“imudp”)
input(type=“imudp” port=“514” ruleset=“remote”)

module(load=“imtcp”)
input(type=“imtcp” port=“514” ruleset=“remote”)

Grafana Alloy then listens for them on port 1514 →
loki.source.syslog “remote_syslog” {
forward_to = [loki.relabel.remote_syslog.receiver]

listener {
address = “0.0.0.0:1514”
labels = {
component = “loki.source.syslog”,
protocol = “tcp”,
job = “va”,
service = “Utilities”,
environment = “prod”,
}
}

listener {
address = “0.0.0.0:1514”
protocol = “udp”
label_structured_data = true
labels = {
component = “loki.source.syslog”,
protocol = “udp”,
job = “va”,
service = “Utilities”,
environment = “prod”,
}
}
}

loki.relabel “remote_syslog” {
forward_to = [loki.write.logs_base.receiver]
rule {
source_labels = [“__syslog_message_hostname”]
target_label = “host”
}

    rule {
            source_labels = ["__syslog_message_severity"]
            target_label  = "level"
    }

    rule {
            source_labels = ["__syslog_message_app_name"]
            target_label  = "application"
    }

    rule {
            source_labels = ["__syslog_message_facility"]
            target_label  = "facility"
    }

    rule {
            source_labels = ["__syslog_connection_hostname"]
            replacement   = "az1-ict-iam-prod-syslog-01.server.ufl.edu"
            target_label  = "connection_hostname"
    }

}

Please advise.

6

Try this, using lok.process instead of loki.relabel:

log “remote_syslog” {
  forward_to = [loki.process.remote_syslog.receiver]

  listener {
    address = “0.0.0.0:1514”
    labels = {
      component = “loki.source.syslog”,
      protocol = “tcp”,
      job = “va”,
      service = “Utilities”,
      environment = “prod”,
    }
  }

  listener {
    address = “0.0.0.0:1514”
    protocol = “udp”
    label_structured_data = true
    labels = {
      component = “loki.source.syslog”,
      protocol = “udp”,
      job = “va”,
      service = “Utilities”,
      environment = “prod”,
    }
  }
}

loki.process "remote_syslog" {
  forward_to = [loki.write.destination.receiver]

  
  stage.static_labels {
    values = {
      connection_hostname = "az1-ict-iam-prod-syslog-01.server.ufl.edu",
    }
  }

  stage.labels {
    values = {
      level = "__syslog_message_severity",
      application = "__syslog_message_app_name",
      facility = "__syslog_message_facility",
    }
  }
)
7

Unfortunately it still does not dynamically label the syslog messages in the same way that static mode does… This is a bit problematic. Any other ideas up your sleeve ?

8

Good afternoon,

I tried this again with syslog-ng instead of rsyslog, it made no difference, I guess I’ll open a github issue.