How do i use LogQL to zero out IP addresses and PII infromation

Am building a dashboard with Loki LoqQL and i need to hide IP addresses and PII from the logs been shown since this servers are in protected Federal space hence need to hide this.
Any help, tried using the regexp and pattern expression but this isn’t working.

Honestly if you are talking federal you should cleanse the logs before they even make it into Loki. You’ll need to do this with a middleware, you can write your own scripts, or there are other open source tools that can do this.

1 Like

any suggestion on third party middlewares.

I’ve used GitHub - microsoft/presidio: Context aware, pluggable and customizable data protection and de-identification SDK for text and images with Kafka before, but I’d imagine with some work you can fit that in somehow in your log pipeline before sending stuff to Loki.

@edemkporha, about third party tools… aren’t you using promtail to send data to Loki? If yes, try to see if a replace on your scrape config would be enough to mask the data.

I am, using promtail to send data to loki , this is on centos server.
can you share the script on how this can be done

Quote from the docs Configure Promtail | Grafana Loki documentation

The replace stage is a parsing stage that parses a log line using a regular expression and replaces the log line.

  # The RE2 regular expression. Each named capture group will be added to extracted.
  # Each capture group and named capture group will be replaced with the value given in
  # `replace`
  expression: <string>

  # Name from extracted data to parse. If empty, uses the log message.
  # The replaced value will be assigned back to soure key
  [source: <string>]

  # Value to which the captured group will be replaced. The captured group or the named
  # captured group will be replaced with this value and the log line will be replaced with
  # new replaced values. An empty value will remove the captured group from the log line.
  [replace: <string>]
1 Like