Top 10 X - LogQL, Promtail and Graphs from Firewalllogs

Grafana Loki
#1

Hello,

another question: I would like to show a Top 10 (or Top 5, Top 15…) list of something from firewall syslog lines. Maybe “Top 10 Destination ports” (see example: …DPT=1900…) or “Top 15 Destination IPs”. Maybe even with hits (in a table or a graph), like:

443 → 150 hits
80 → 101 hits
22 → 80 hits
…

This is an example Log Line:

[LAN_LOCAL-default-A]IN=eth1.1 OUT=eth1.2 MAC=ff:ff:ff:ff:ff:ff:dc:b6:33:41:a4:52:07:01:55:10:01:41 SRC=192.168.0.2 DST=255.255.255.255 LEN=147 TOS=0x00 PREC=0x00 TTL=64 ID=4795 DF PROTO=UDP SPT=53502 DPT=1900 LEN=127

I tried stuff like:

topk(10,sum(rate(({__error__ != "",job="syslog"} | logfmt)[16h])) by (DPT))

topk(10,sum(rate(({job="syslog"} | logfmt)[5m])) by (DPT))

sum(rate(({job="syslog"}|logfmt)[10s])) by (DPT)

sum(count_over_time({job="syslog", host="blackhole"}[1s] |= "2013-D]" | logfmt)) by (DPT)

But I can’t get it to work. Either I get an error, or I get no Data, or I get a graph with just one line and without a name in legend (just { }).
I used “logfmt”, because I only get “host” and “job” as a lable. I thougth, if I want to group something (… by XXX) it must a lable and not only a “detected field”.

Maybe someone could enlighten me.

Systeminfo:

  • Raspberry Pi 4 Model B Rev 1.2 (4GB, arm, non-64bit)
  • Raspbian GNU/Linux 10 (buster)
  • Docker version 20.10.5, build 55c4c88
  • Grafana v7.5.2 (ca413c612f)
  • Promtail: Version=master-4d1da2e, branch=master, revision=4d1da2ed
  • Loki Version unkown (should be a couple of days old, with docker pull)
  • Logs are received from a local rsyslog (which receive them from the firewall)

I’m starting the “whole Grafana Stuff” with docker-compose.yml:

version: "2"
services:
  influxdb:
    image: influxdb:1.8 #armv7
    container_name: influxdb
    restart: always
    ports:
      - "8083:8083"
      - "8086:8086"
      - "8090:8090"
    env_file:
      - 'env.influxdb'
    volumes:
      - ./influxdb/data:/var/lib/influxdb

  telegraf:
    image: telegraf:latest
    container_name: telegraf
    restart: always
    links:
      - influxdb
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro

  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-local-config.yaml:/etc/loki/local-config.yaml:ro
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    ports:
      - "1514:1514"
    volumes:
      - ./promtail:/var/log
      - ./promtail-docker-config.yaml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    ports:
      - "3001:3000"
    env_file:
      - 'env.grafana'
    user: "0"
    links:
      - influxdb
    volumes:
      - ./grafana/data:/var/lib/grafana

Thank you so much in advance.

Regards,
Jens

#2

bump

No one? Is this the wrong place for this kind of question or did I missed to provide some other important informations?