Rsyslog/promtail/loki/grafana help

Grafana Loki
, , , ,
1

I’ve been working on testing out a grafana/loki/promtail/rsyslog setup for my home environment and I’ve got it at least setup and working. However, I wanted to be able to clearly identify the log message so that I can further breakdown the different fields in the messages (and then be able to query/create dashboards based on those fields). For example, a PfSense log, I want to see the full message and then be able to break it down by the various fields <Pass/Block>, etc, and of course be able to do similar for logs from NAS and other systems. I’m not fully sure if this is something done at query time in Grafana or something changed in the config(s). I may have overlooked something that might be right in my face but I have been having trouble the last few days trying to figure this out, any help is greatly appreciated

I followed the documentation on these two links:

and Promtail - rewriting output (and using internal labels) - #2 by wrightsonm

to get it to where it is now.

Here is an example picture of how the log looks now in grafana:

image
image1773×270 62.1 KB

My promtail-local-config.yaml file:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: lokiSystem
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:1514
    labels:
      job: syslog
  relabel_configs:
    - source_labels: [__syslog_message_hostname]
      target_label: host
    - source_labels: [__syslog_message_hostname]
      target_label: hostname
    - source_labels: [__syslog_message_severity]
      target_label: level
    - source_labels: [__syslog_message_app_name]
      target_label: application
    - source_labels: [__syslog_message_facility]
      target_label: facility
    - source_labels: [__syslog_connection_hostname]
      target_label: connection_hostname

my loki-local-config.yaml file:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

and lastly, my rsyslog.conf file edited with the portion to add from the first link:

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

# Forward everything
*.*  action(type="omfwd"
       protocol="tcp" target="127.0.0.1" port="1514"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")

# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

Any help would be greatly appreciated.

2

Here is the general recommends when it comes to what should or should not be made into labels: Label best practices | Grafana Loki documentation

Fields such as block/pass or protocol (tcp/udp) make good labels, while fields such as source IP / destination IP / MAC address don’t.

In terms of how to process your pfsense log, first you’d need a way to identify it. If your pfsense host can be identified via hostname, you can use that. Or I see you have an application field as filterlog, assuming that’s what your pfsense logs come in as you can use that as well. If neither fits you can add an identifying label in your syslog template.

Once you have a way to identify your pfsense logs from the rest, you just need a match block and add your processing in that block:

- match:
    selector: '{application="filterlog"}' # or '{hostname="pfsense"}', or other label
    stages:
      # pfsense specific processing happens here
3

Hi there, thanks for your reply! I have read that about the labels already so that makes sense. I dont necesarily want to make them into ‘labels’ as we are thinking here, but I suppose for lack of a better term - fieldnames? so could easily pick out protocol, pass/block, source/dest IP, etc. I think you understand what I’m getting at based on your reply. Yes the logs can be identified by hostname already and yes the filterlog is how the pfsense logs come in (for the firewall logs at least). but again what I want to do is from there (pfsense filterlog for example) is grab those fields like I mentioned already.
So it seems then my configs are right, I would just need a match and processing block and could set it up that way, and could do similar for other host logs as well and process them differently. I think that makes sense. I’ll just have to read up on how to achieve that processing then. Thank you!

edit: “If neither fits you can add an identifying label in your syslog template.” I’m sure tackling this might be different from what I was trying but I also tried using a custom syslog template set how I liked it but then promtail complained due to missing pri and then a version number

4

No problem. If you can provide an example log and which fields you want to extract as labels I can give you an example configuration.

5

Absolutely! Here is one I just grabbed very quickly:
115,1000006961,mvneta0,match,pass,out,4,0x0,63,5593,0,DF,6,tcp,60,96.244.89.201,192.0.73.2,24203,443,0,S,3959689068,64240,mss;sackOK;TS;nop;wscale

I’d like the ‘mvneta0’ to be “Interface”, ‘pass’ to be “Action”, ‘out’ to be “Direction”, ‘4’ to be “IP Version”, ‘tcp’ to be “Protocol”, ‘96.244.89.201’ to be “Source IP”, ‘192.0.73.2’ to be “Dest IP”. I’m sure there are others too but I’d need to remember what the other fields represent, not sure off the top of my head. Once I see how it can be done though I’m sure I could tackle them.

edit: You mentioned so they could be extracted as labels. Is this the only way to achieve what I’m thinking about? Just curious as I know from what I’ve seen online you dont want a whole lot of different labels. I just want to easily be able to see and differentiate these fields and then be able to query the pfsense filterlog for “Action=block” for a simple example

6

Something like this should work (not tested):

- match:
    selector: '{application="filterlog"}' # or '{hostname="pfsense"}', or other label
    stages:
      - regex:
          expression: `^[^,\s]+,[^,\s]+,(?P<interface>[^,\s]+),[^,\s]+,(?P<action>[^,\s]+),[^,\s]+,(?P<ip_version>[^,\s]+),[^,\s]+,[^,\s]+,[^,\s]+,[^,\s]+,[^,\s]+,[^,\s]+,(?P<protocol>[^,\s]+),.+`
      - labels:
          interface:
          action:
          ip_version:
          protocol:

Source and destination IPs aren’t good candidates for labels, so I’d recommend you to parse those at query time with something like pattern filter. You can play around with it here: LogQL Analyzer | Grafana Loki documentation

7

I think what I may be looking for then might actually be the parsing at query time with the pattern filter. I just wasnt sure if it could have been done somewhere else (without making them labels necesarily since I want to avoid going against best practice). However, this is great, I’ll have to give this a test and play around with this and the pattern queries. Thank you so much. I’ve been banging my head trying to figure this out for days now and you’ve been a huge help! I had no idea that query analyzer existed either so thats amazing.

8

I’m having trouble with the provided config. I’ve got it added to the promtail config file, but it appears that nothing has changed in grafana after editing the promtail config and restarting the service. I’m curious if its not picking up the actual message part of the log and is maybe trying to apply the regex expression to the WHOLE log entry (quite possibly my fault since I was thinking only the actual message portion of the log)? I’m not terribly sure as to why, just a guess from my end. I dont see any errors in rsyslog, promtail, or loki.

edit:
I think that might be what is happening. I looked in the syslog logs and there is a timestamp, then the hostname, followed by the application, then process ID, then it gets to the actual log (what was provided above). So the whole log that gets sent to promtail would look like the following:
Dec 5 13:05:31 pfSense.home.arpa filterlog[24343] 113,1000006865,mvneta1,match,pass,out,4,0x0,127,29976,0,none,17,udp,76,10.10.20.3,10.10.10.5,55644,53,56

But what is strange (to me at least) is I’d think the regex filtering would still populate some of those labels with SOMETHING from the full log (could be wrong - I’ve never been good with regex…), but the log lines dont look any different in grafana at all. Continuing to look into this on my end. Is there a way to grab the log message in promtail so that I could apply the regex strictly to that, or would it have to be applied to the FULL log?