I am pushing syslog log to Promtail and visualise it in Grafana.
It all works in principle, but in Grafana the log lines do not contain the name of the host (source of the log event) not the name of the application i.e. a line that looks like
Apr 22 22:10:01 gytha CRON[1570974]: (root) CMD (/opt/prometheus/bin/pushDNSPing)
in syslog, shows up as
2024-04-22 22:10:02.184 (root) CMD (/opt/prometheus/bin/pushDNSPing)
in Grafana (when option Time is enabled in Grafana) i.e. in Grafana the host name (gytha) and the application/process (CRON[1570974]) are missing.
I have tried many different approaches trying to piece the data together, but was not able to make the lines in Grafana look like in syslog files, but no lock.
The closest I came too success was parsing the message contents using regex, use a relabel_config to concatenate fields and output to create the final message, but the label created in the relabel_config never contained the parsed message, even though it was set correctly as label. The corresponding config looks like this:
scrape_configs:
- job_name: syslog
syslog:
labels:
job: syslog
listen_address: 0.0.0.0:9081
use_incoming_timestamp: true
relabel_configs:
- source_labels: [__syslog_message_app_name]
target_label: application
- source_labels: [__syslog_message_facility]
target_label: facility
- source_labels: [__syslog_message_hostname]
target_label: host
- source_labels: [__syslog_message_severity]
target_label: level
- source_labels: [__syslog_connection_hostname]
target_label: remote_hostname
- source_labels: [__syslog_connection_ip_address]
target_label: remote_ip
- source_labels: ['host', 'application', 'message']
separator: '-'
target_label: 'formatted_message'
pipeline_stages:
- match:
selector: '{job="syslog"}'
stages:
- regex:
expression: '^\s*(?P<message>.*)[\r\n]*$'
- labels:
message:
- output:
source: formatted_message
And for this I config, Grafana contains
As one can see, event though label message appears correctly in Grafana, it is not part of label formatted_message.
I tried different orders etc., but nothing helped.
How can the original syslog format be reconstructed in Grafana?
