Promtail/Loki/Grafana GeoIP questions

Hello,

I’ve successfully been able to get the GeoIP promtail pipeline stage working against the source IPs for my pfSense firewall logs which raised two questions for me:

1- Is it possible to do geolocation for both source IPs and destination IPs?
2- If the above is possible, is it possible to, (and if so) how could I rename the source IP geolocation labels so that I could differentiate between geolocation data for the source IPs and destination IPs?

Additionally, is it possible to do accomplish the geolocation data at query time rather than as a promtail stage? I just was thinking if this is possible, it may be easier to accomplish the above this way maybe (as well as less promtail processing, even if very minimal hit due to this).

Thanks!

I just asked this question to Grot (the AI bot on Grafana’s site). I’m not very technical so I hope it helps.

  1. Yes, it is possible to perform geolocation for both source IPs and destination IPs in Promtail. You can configure multiple instances of the geoip pipeline stage in your Promtail configuration to perform geolocation lookups for both source and destination IPs. Each instance of the geoip stage can have its own set of labels to store the geolocation data.

For example, you can configure one geoip stage to perform a lookup on the source IP and add labels like source_geoip_country_name, source_geoip_city, etc. Similarly, you can configure another geoip stage to perform a lookup on the destination IP and add labels like destination_geoip_country_name, destination_geoip_city, etc.

  1. To differentiate between geolocation data for source IPs and destination IPs, you can use different label names for each type of IP. For example, you can use source_geoip_country_name and destination_geoip_country_name to store the country names for the source and destination IPs, respectively. Similarly, you can use different label names for other geolocation data like city, region, etc.

By using distinct label names, you can easily differentiate between the geolocation data for source and destination IPs when querying and visualizing the data in Grafana.

Regarding your question about performing geolocation data at query time instead of as a Promtail stage, currently, Promtail does not have built-in functionality to perform geolocation lookups at query time. The geolocation data is typically added as labels during the log ingestion process by Promtail. However, you can use Grafana’s transformation capabilities, such as the lookup transformation, to enrich your log data with geolocation information at query time. This would involve setting up a lookup table or external data source that maps IP addresses to geolocation data and using the lookup transformation in your Grafana queries to retrieve the geolocation information based on the IP addresses in your logs.

Keep in mind that performing geolocation lookups at query time may introduce additional overhead and latency compared to adding the geolocation data during log ingestion with Promtail. It’s important to consider the trade-offs and performance implications when deciding between these approaches.

1 Like

No worries, this was definitely helpful. Now I know its at least possible to do what I’m thinking. just need to figure out how to implement it now.

1- I tried a few things trying to change the geoIP labels but maybe I just hadnt found the right thing to try yet.

2- Makes perfect sense now that I know the above is possible. This part should be easy once I can figure out #1.

Regarding the geolocation at query time - So it sounds like its possible to achieve this through the lookup tables then, just maybe something thats not done this way very often. I just wasnt sure if it may be better to handle at query time or not (and based on my previous research it seems at least for adding labels and what not, its better to handle certain things at query time).

Great information. Thank you.

1 Like

If anyone can give me some pointers on how to accomplish the renaming of the geoip labels that would be greatly appreciated as I’m struggling to figure it out. I’ve also asked Grot a couple times as well and have tried things like:

      - geoip:
          db: "/mnt/config/GeoLite2-City.mmdb"
          source: "ip"
          db_type: "city"
          target_labels:
            - custom_city_label: geoip.city.names.en
            - custom_country_label: geoip.country.names.en

and

      - geoip:
          db: "/mnt/config/GeoLite2-City.mmdb"
          source: "ip"
          db_type: "city"
          target_labels:
            - city_name
            - country_name

but havent had any luck so far…

after multiple attempts and questions to GROT, I’m not so convinced that this is actually possible in promtail. If anyone knows otherwise please let me know.
Thanks

I found a way (sort of). I can set a stage to run geoip on source IP for pfsense firewall logs with a direction of “in” and then do the same for destination IP with a direction of “out” which sort of accomplishes what I was looking for. Ideally I wanted to be able to have both for both directions but I got this to work and shows the geoip information for source IPs for stuff coming in and then does geoip information for destination IPs for stuff going out. So sort of a work around but at least got this to work. Still havent found a way to rename the labels but doing it this way doesnt really matter as much.