Hi there,
#config.alloy
logging {
level = "debug"
format = "logfmt"
}
loki.source.syslog "syslog" {
listener {
address = "0.0.0.0:1514"
protocol = "tcp"
use_rfc5424_message = true
}
forward_to = [loki.write.grafana_loki.receiver]
}
loki.write "grafana_loki" {
endpoint {
url = "http://loki.intern/loki/api/v1/push"
}
}
When sending data to alloy then this message is appearing in the logs:
ts=2025-03-01T17:20:57.011031542Z level=error msg="final error sending batch" component_path=/ component_id=loki.write.grafana_loki component=client host=loki.intern status=400 tenant="" error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"
I tried to add to config.alloy:
loki.source.syslog "syslog" {
...
labels = {
env = "production"
}
}
But this does not work either.
How do i solve this?
What is your configuration with labels looking like?
Loki is in General using Port 3100 and you do Not have this in your loki.write. is this expected?
Yes that’s expected, no port needed as i point the default port 80
Check this how to Labelmap Labels and the possible Bug.
opened 10:31PM - 11 Dec 24 UTC
bug
### What's wrong?
loki.source.syslog should provide __syslog_message_* internal… labels which if needed can be relabeled into static labels.
However with grafana alloy 1.5.1 this is not working and you have to use a workaround.
### Steps to reproduce
The important part in this config is this: You ne a loki.relabel with action = labelmap.
```
loki.relabel "syslog_01" {
rule {
action = "labelmap"
regex = "__syslog_(.+)"
}
```
Second and most important ist you need to address the component as:
` relabel_rules = loki.relabel.syslog_01.rules`
and NOT as
` forward_to = [loki.relabel.syslog_01.receiver]`
```
loki.source.syslog "syslog_receiver_udp_514" {
// RFC5424 listeners on port 514
listener {
address = "0.0.0.0:514"
protocol = "udp"
use_incoming_timestamp = false
label_structured_data = true
use_rfc5424_message = true
max_message_length = 0
syslog_format = "rfc5424"
labels = {
protocol = "udp",
format = "rfc5424",
port = "514",
service_name = "syslog_rfc5424_514",
}
}
relabel_rules = loki.relabel.syslog_01.rules
// forward_to = [loki.relabel.syslog_01.receiver]
forward_to = [loki.process.syslog_processor.receiver]
}
loki.source.syslog "syslog_receiver_tcp_514" {
listener {
address = "0.0.0.0:514"
protocol = "tcp"
use_incoming_timestamp = false
idle_timeout = "120s"
label_structured_data = true
use_rfc5424_message = true
max_message_length = 0
syslog_format = "rfc5424"
labels = {
protocol = "tcp",
format = "rfc5424",
port = "514",
service_name = "syslog_rfc5424_514",
}
}
relabel_rules = loki.relabel.syslog_01.rules
// forward_to = [loki.relabel.syslog_01.receiver]
forward_to = [loki.process.syslog_processor.receiver]
}
loki.source.syslog "syslog_receiver_udp_5514" {
// RFC3164 listeners on port 5514
listener {
address = "0.0.0.0:5514"
protocol = "udp"
use_incoming_timestamp = false
max_message_length = 0
syslog_format = "rfc3164"
labels = {
protocol = "udp",
format = "rfc3164",
port = "5514",
service_name = "syslog_rfc3164_5514",
}
}
relabel_rules = loki.relabel.syslog_01.rules
// forward_to = [loki.relabel.syslog_01.receiver]
forward_to = [loki.process.syslog_processor.receiver]
}
loki.source.syslog "syslog_receiver_tcp_5514" {
listener {
address = "0.0.0.0:5514"
protocol = "tcp"
use_incoming_timestamp = false
idle_timeout = "120s"
max_message_length = 0
syslog_format = "rfc3164"
labels = {
protocol = "tcp",
format = "rfc3164",
port = "5514",
service_name = "syslog_rfc3164_5514",
}
}
relabel_rules = loki.relabel.syslog_01.rules
// forward_to = [loki.relabel.syslog_01.receiver]
forward_to = [loki.process.syslog_processor.receiver]
}
loki.relabel "syslog_01" {
rule {
action = "labelmap"
regex = "__syslog_(.+)"
}
rule {
source_labels = ["__syslog_message_hostname"]
target_label = "instance"
}
rule {
source_labels = ["__syslog_message_severity"]
target_label = "level"
}
rule {
source_labels = ["__syslog_message_app_name"]
target_label = "application"
}
rule {
source_labels = ["__syslog_message_facility"]
target_label = "facility"
}
rule {
source_labels = ["__syslog_connection_hostname"]
target_label = "connection_hostname"
}
rule {
source_labels = ["__syslog_message_proc_id"]
target_label = "proc_id"
}
rule {
source_labels = ["__syslog_message_msg_id"]
target_label = "msg_id"
}
rule {
source_labels = ["__syslog_connection_ip_address"]
target_label = "connection_ip_address"
}
forward_to = [loki.process.syslog_processor.receiver]
}
loki.process "syslog_processor" {
forward_to = [loki.write.loki.receiver]
// Add specific syslog fields to structured metadata
stage.structured_metadata {
values = {
level = "__syslog_message_severity",
severity = "__syslog_message_severity",
facility = "__syslog_message_facility",
hostname = "__syslog_message_hostname",
instance = "__syslog_message_hostname",
app_name = "__syslog_message_app_name",
proc_id = "__syslog_message_proc_id",
msg_id = "__syslog_message_msg_id",
connection_ip_address = "__syslog_connection_ip_address",
connection_hostname = "__syslog_connection_hostname",
}
}
// Keep only the message content
stage.output {
source = "__syslog_message"
}
}
```
This issue was shortly discussed in slack:
https://grafana.slack.com/archives/C01050C3D8F/p1733952739924029?thread_ts=1733928788.934089&cid=C01050C3D8F
PS:
You should update the documentation of loki.source.syslog and list all the available internal labels which can be used.
Add an example how to remove the labels and add it as structured_metadata to avoid high cardinality.
### System information
Linux ubiquiti 5.10.0-33-amd64 #1 SMP Debian 5.10.226-1 (2024-10-03) x86_64 GNU/Linux
### Software version
Grafana Alloy 1.5.1
### Configuration
```text
```
### Logs
```text
```
That link had helped, Thank you!
1 Like
