Grafana geomap from loki line format data

Grafana Loki
,
1

Hi,
I am trying to make a geomap panel in grafana. My setup (but not the problem) looks very similar to Create worldmap or Geomap with Loki Geoip

The most visible difference to the setupi in this question I see is, that my log data is not in json, but in syslog format and that my query hence doesn’t use “|json”. I am not sure if and what implications that may have for the data returned from the query.

I have tried with grafana 9.5.8 (most of the time) and grafana 10.1.0.

promtail 2.8.6 configuration:

  - match:
     selector: '{job="login",unit="kernel",host=~"login.*|vis.*|adm.*"} |= "FINAL_REJECT"'
     pipeline_name: "GeoIPFirewallReject"
     stages:
     - regex:
        expression: ".*SRC=(?P<ip>[^ ]+) .*"
     - geoip:
        db: "/var/lib/loki/GeoLite2-City_20231117/GeoLite2-City.mmdb"
        source: "ip"
        db_type: "city"
     - labelallow:
       - host
       - job
       - priority
       - unit
       - facility
       - geoip_city_name
       - geoip_country_name
       - geoip_location_latitude
       - geoip_location_longitude

Log lines look something like

Nov 21 12:33:16 login03 kernel: FINAL_REJECT: IN=bond0 OUT= MAC=b8:etc SRC=10.100.100.10 DST=10.100.100.100 LEN=52 TOS=0x00

The query used in Grafana is e.g. (I tried more than one query):

sum by (geoip_location_latitude,geoip_location_longitude) (count_over_time( {job="justus2",host=~"login.*|vis.*|adm.*",unit="kernel",geoip_city_name!=""} |= "FINAL_REJECT" [1h]))

The problem is, that grafana identifies the whole label-set as a “field” for my choice of langitude/latitude fields.

I tried the “labels to fields” transform and many more different combinations of transforms, but have not been able to generate something that the geomapdata panel is willing to parse and display.

The pictures shows the selection I get when trying to choose the “Latitude field”.

geomap

Querying without count_over_time() and sum() yields the expected results with the needed tags present.

e.g. (copied from grafana explore results without functions):

(log line) 
Fields: 
facility:kern 
geoip_city_name:Augsburg 
geoip_country_name:Germany 
geoip_location_latitude:48.3506 
geoip_location_longitude:10.9114

Thanks in advance.

2

Hi,

You can try the extract fields transformation, to break your data containing your latitude and longitude into separate fields, and then you can change the location mode to coords, and select the fields that were separated out.

3

Hi,

I had tried the “extract fields” transformation. It shows the same problem, as the map plugin, in that it gives me “fields” to choose from, which contain the whole labelset.

grafana_extract_fields
grafana_extract_fields732×323 31 KB

Ofc, I can choose “time” or “value”, as source and “replace all fields”… but that yields no usable fields either.

Hmm, I just noticed I can reduce the many fields to one by setting the legend-name under “options” in the query - then if I just give that the name “results”, this is the one field I can choose.

Also, I thought the “Labels to Field” transformation would be exactly what I need…

Oh, I think I figured it out:

grafana_geoquery_working
grafana_geoquery_working883×394 39.2 KB

grafana_geoquery_working1

  • set the legend-name in “options” to some string
  • use “instant” as query type
  • use “labels to fields” (only works after legend-name is set to something fixed, else you can select only one labelset of the many)
  • convert field type (thanks to the other discussion)

Cheers

PS. wishing it would be more obvious what data a plugin/panel expects, how my data differs from it and what transforms actually do with the data.
I know I can klick on the “bug” in transforms and use the query inspector, but the process is still very far from straight-forward

PPS. A bit astonished, that geoip data from logs took so long to figure out. I would have thought that is a very common thing independent of the type of log data you have. I still don’t know why it worked in the other thread with json log data.

2 Likes