Recommended way to Geo locate IP's for heatmap

Grafana Loki
1

Hey,

I have a working setup of Telegraf / Promtail running on Ubuntu 20.04 to send metrics and apache access logs to Grafana & Loki. Metrics and logs are being ingested fine.

I’m looking to add a map based heatmap of IPs from the access logs - but googling seems to just bring up lots of bespoke scripts or a telegraf processor that requires building from source on the server (a bit beyond me, and not sure i want to run that on a production web server).

Is there a “proper” way to do this within Loki or Grafana, or is the only option to write a custom script to pull IP and write out lat / lon ?

I’m fairly new to Loki so may have missed something fundamental but i can’t find a way to do this “in app” as it were.

Any advice you can offer would be greatly appreciated

2

Welcome,

So if you have the whole pipeline working could you please post sample logs here so we can see how to use grafana to extract the long lat.

image
image979×721 30.1 KB

3

image
{filename=“/var/log/apache2/access.log”} | pattern <ip> - - <_> "<method> <url> <_>" <status> <_> <_> "<user_agent>" <_>

Is this enough information?

Also, to be clear i need to convert to lat lon - i dont have those yet, this is the part i’m stuck on.

4

Would be nice if you could fill out the missing data with obfuscated/bogus data instead of - -. the more realistic data you provide the easier to help. by realistic I do not mean real live data. So where is the long lat data you want to marry to this log file? and would the linking data piece be ip address?

5

I get the impression that the OP’s fundamental problem / requirement is
working out how to convert an arbitrary IP address into a geographical
location.

Antony.

1 Like
6

gotcha! some sort of IP Geolocation?

7

Hey - so there is no missing data - that is my raw query in grafana. The <_> is just me ignoring certain parts of the log line and - - is actually present in the log file as per;

image
image1467×487 74.3 KB

Antony is correct - i need to do the geolocation to convert into lat long, i’ve used the worldmap with a gps tool before so not worried about that side.

Do i need to write a script to pull data from influxdb and query an external api for every IP or is there some functionality within loki or grafana that will allow me to do this?

1 Like
8

I am not sure on how to do it from grafana or Loki. And I would not recommend doing it dynamically neither as that would be costly on performance.

Your best route might be like you said writing something in your telegraf, that would call the ip to geo api, then writes it to influxdb.

I am not sure it is doable “in app”

9

unless you might be able to do the following for starters to investigate the feasability.

make the data a dashboard variable, selecting only the ip field and then feed that variable to the ip to geolocation endpoint. test it out with one ip address and then see if there is an api that takes an array or dict or ip addresses. then plot on geomap?

10

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.