How to parse a log line into something useful?

[lonelyadmin]

Noob here with Loki. I’m getting logs into loki like this:

772 <6> 2019-12-26T10:19:09Z ip-100-69-22-95.us-east-2.compute.internal ESS101777[1]:
{“message_id”:”1577355532-893278-20131-541688-1”,”src_ip”:”93.99.104.21”,”hdr_from”:”MAILERDAEMON@localhost (Mail Delivery
System)”,”account_id”:”ESS101777”,”domain_id”:”205534”,”ptr_record”:”emkei.cz”,”attachments”:null,”recipients”:[{
“action”:”allowed”,”reason”:”recipient”,”reason_extra”:”default_scan_policy:exempt”,”delivered”:”delivered”,”deliver
y_detail”:”smtp.secureserver.net:25:250 2.0.0 kQEHil5TWH1uq – kQEHil5TWH1uqkQEIitWFp mail accepted for
delivery”,”email”:”info@connect-ag.in”}],”hdr_to”:”info@connect-ag.in”,”recipient_count”:1,”dst_domain”:”connectag.in”,”size”:3566,”subject”:”Undelivered Mail Returned to Sender”,”env_from”:””,”timestamp”:”2019-12-
26T10:18:54+0000”}

It looks like part json and part crap at the beginning I don’t care about.

I don’t have control over the source, but I do control everything else.
3rd party cloud appliance -> my syslog-ng -> promtail/loki -> grafana

How can I get the json bit parsed into something useful? I’m wanting to report/graph based on the key:values within the {}.

Thanks!

[lonelyadmin]

Ok, getting somewhere through trial and error. Using the config below, I’m able to get only the json into loki. So now parseable fields come up. Problem is in the loki query, the part I need to match against is in a key-value pair and I can’t figure out how to drill down into it.

Do I need to add another pipeline in the config like from here?

Trying to match against some of the fields in “recipients”, mostly “action” and “delivered”.

”recipients”:[{
“action”:”allowed”,”reason”:”recipient”,”reason_extra”:”default_scan_policy:exempt”,”delivered”:”delivered”,”deliver
y_detail”:”smtp.secureserver.net:25:250 2.0.0 kQEHil5TWH1uq – kQEHil5TWH1uqkQEIitWFp mail accepted for
delivery”,”email”:”info@connect-ag.in”}]

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: syslog
  pipeline_stages:
  - replace:
      expression: '(<\d> \d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\dZ (?:[_a-z0-9](?:[_a-z0-9-]{0,61}[a-z0-9])?\.)+(?:[a-z](?:[a-z0-9-]{0,61}[a-z0-9])?) ESS\d\d\d\d\d\[\d\]: )'
        #  - json:
        #expressions:
        #user:
        #source: recipients
  syslog:
    listen_address: localhost:1514
    idle_timeout: 60s
    label_structured_data: yes
    labels:
      job: "syslog"
  relabel_configs:
  - source_labels: ['__syslog_message_hostname']
    target_label: 'host'

- job_name: system
  static_configs:
  - targets:
      - 127.0.0.1
    labels:
      job: varlogs
      __path__: /var/log/*log