Parse and Visualize PFsense Logs

Grafana
, ,
1

I am very new to the grafana and logging ecosystem ive been able to setup log sending but I have no idea how to get the logs visualized. There seems to be no good tutorials/documentation to help do what i would like to do.

I am trying to display the top ten most common source IP that shows up in my firewall logs. I was trying to build off of that but nothing including using 3 gpt apps couldnt help me. Ive tried regex, ive tried different graphs but i cannot get to the end result. I can get the regex to label the source ip but i cannot visualize the top ten most common ips. I want to do more complex stuff but cant even do this.

Pfsense firewall log

2024-12-06T19:52:25.669653-08:00 test.test.test filterlog 77859 - - 4,,,1000000103,em0,match,block,in,4,0x0,,53,0,0,DF,17,udp,61,157.240.22.19,72.203.171.222,443,13504,41

promtail yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Job for system logs (default system logs like syslog, auth.log, kern.log, etc.)
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: system
          __path__: /var/log/syslog

  # Job for syslog-ng logs forwarded via unique port (e.g., 10514)
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:5514  # Listening for syslog messages on port 5514
      listen_protocol: tcp  # Using TCP for syslog messages
      idle_timeout: 60s  # Timeout after 60 seconds of inactivity
      label_structured_data: yes  # Include structured data from syslog header as labels
      labels:
        job: "syslog"  # Constant label for all syslog entries
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'  # Relabel syslog hostname field as "host" 
    pipeline_stages:
      - regex:
          expression: '^\\S+\\s+\\S+\\s+\\S+\\s+\\S+\\s+-\\s+-\\s+\\S+,,,.*,(?P<interface>\\S+),.*,(?P<action>block|pass),.*,(?P<ip_version>\\d+),.*,(?P<protocol>\\S+),.*,(?P<source_ip>\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}),(?P<destination_ip>\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}),.*'
      - labels:
          interface:
          action:
          ip_version:
          protocol:
          source_ip:
          destination_ip:

loki yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_line_size: 256KB
  max_streams_per_user: 10000
  max_global_streams_per_user: 20000
  reject_old_samples: true
  reject_old_samples_max_age: 24h
  max_query_series: 10000

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false