Promtail Use Case: Drop anything except lines matching a Regex

Grafana Loki
1

Hello Community,

I have a legacy system which generates enormous amounts of logs. In order to get this system attached to Loki my idea is to have a configuration that drops anything per default except lines that match a Regex ruleset. The only thing I found is the drop Stage but this is the opposite I want. Is my use case feasible with Promtail? If not, which log-shipper may I use for?

Thanks in advance.

2

Hey @royboerner

Can you invert your regex?

3

Have you looked at the match stage?

4

The drop stage should work for your usecase. define a tag that gets filled when your regex hits and afterwards drop all lines where this tag is empty.

5

Thank you guys.

I can now share a working configuration.

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  #log_level: debug

positions:
  filename: /tmp/promtail-positions.yaml

client:
  url: http://vm061:3100/loki/api/v1/push

scrape_configs:
 - job_name: mes
   static_configs:
   - labels:
      # A `job` label is fairly standard in prometheus and useful for linking metrics and logs
      job: mes
      # A `host` label will help identify logs from this machine vs others
      host: ${HOSTNAME:localhost}
      # The path matching uses a third party library: https://github.com/bmatcuk/doublestar
      __path__: /app/300works/log/MES_PROD.{FABsrv,TRANSsrv,EDCsrv,EPRsrv,LISTsrv,OCAPsrv,QUERYsrv,RTLsrv}_log
   pipeline_stages:
   - match:
       selector: '{job="mes"}'
       stages:
       - regex:
           expression: '^(?P<timestmap>\d{2}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}.\d{2}).*$'
       - timestamp:
           source: timestmap
           # 09/21/21 13:09:38.73
           format: '01/02/06 15:04:05.999'
       - regex:
           expression: 'fwsrvExecServerRule:start executing rule (?P<rule_name>.*)$'
       - labels:
           rule_name:
       - regex:
           expression: 'FwTransaction::storeIt:(?P<rule_name>[^\s]+)'
       - labels:
           rule_name:
       - regex:
           expression: 'FwTransaction::doIt:(?P<rule_name>[^\s]+)'
       - labels:
           rule_name:
       - regex:
           expression: 'FwDbStmtOracle::execute:(?P<rule_name>[^\s]+)'
       - labels:
           rule_name:
   - match:
       # drop anything that has no rule_name label
       selector: '{job="mes", rule_name!~"(.+)"}'
       action: drop
3 Likes
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.