How should JSON logs be parsed in Loki using Promtail?

[imdkamrul1995]

I’m currently facing difficulties configuring Promtail to parse JSON logs in Loki successfully. The log structure consists of a plain JSON string without any nesting

{ LEVEL: INFO,Class: net.provisioning.controller.BarringController,Thread: http-nio-9720-exec-4,IP: bs591/127.0.1.1,host: bs591,message: POST /barring/barrings-with-level,requestParam: actionType=ddd&channel=sss&reseller=ss&salesman=ss,requestBody: {“subscriptionId”:0,“channel”:“string”,“barringType”:“string”,“reseller”:“string”,“salesman”:“string”,“barringLevel”:“strinbffgfgfgfgg”},requestHeader: {“sec-fetch-mode”:“cors”,“content-length”:“161”,“referer”:“http://localhost:9720/swagger-ui/“,“sec-fetch-site”:“same-origin”,“accept-language”:“en-US,en;q=0.9”,“cookie”:“Idea-d9de4e9c=d62c8f22-b776-45ab-8abc-54bac78194b7”,“origin”:“http://localhost:9720”,“accept”:“application/vnd.-v1.0+json”,“sec-ch-ua”:”\“Not.A/Brand\”;v=\“8\”, "Chromium";v="114", "Google Chrome";v="114"”,“sec-ch-ua-mobile”:“?0”,“sec-ch-ua-platform”:“"Linux"”,“host”:“localhost:9720”,“connection”:“keep-alive”,“content-type”:“application/json”,“accept-encoding”:“gzip, deflate, br”,“user-agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36”,“sec-fetch-dest”:“empty”},responseHeader: ,responseBody: invalid request properties,responseStatus: 400 BAD_REQUEST,requestId: null }"

I used following promtail config:

server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:

• url: http://localhost:6350/loki/api/v1/push
  scrape_configs:
• job_name: app-provisioning
static_configs:
  • targets:
    • localhost
      labels:
      job: test-app-provisioning
      app: test-app
      host: test-apihub-prod-app
      path: /home/apihub/log/app.provisioning.202311151.json
      pipeline_stages:
    • cri: {}
    • match:
selector: ‘{job=“test-app-provisioning”}’
stages:
- json:
expressions:
level: “LEVEL”
class: “Class”
thread: “Thread”
ip: “IP”
host: “host”
message: “message”
requestParam: “requestParam”
requestBody: “requestBody”
requestHeader: “requestHeader”
responseHeader: “responseHeader”
responseBody: “responseBody”
responseStatus: “responseStatus”
requestId: “requestId”
source: “message”
drop_malformed: false
- labels:
level: “{{ .level }}”
class: “{{ .class }}”
thread: “{{ .thread }}”
ip: “{{ .ip }}”
host: “{{ .host }}”
message: “{{ .message }}”
requestParam: “{{ .requestParam }}”
requestBody: “{{ .requestBody }}”
requestHeader: “{{ .requestHeader }}”
responseHeader: “{{ .responseHeader }}”
responseBody: “{{ .responseBody }}”
responseStatus: “{{ .responseStatus }}”
requestId: “{{ .requestId }}”
      I expected that by specifying the JSON parsing and labels, I would be able to view these labels in Grafana Explore. While I can see the log entries in Grafana, as depicted below, the issue is that I’m not seeing any labels that correspond to the JSON keys

Any guidance on how to make this configuration work would be greatly appreciated. Thanks for any assistance!

[b0b]

I would recommend reading this The concise guide to labels in Loki | Grafana Labs

For best Loki performance, you want as few labels as possible. You would ususaly parse the logs at query time.

[tonyswumac]

1. Just to echo b0b’s comment, please read the documentation linked and put those into your consideration when deciding what fields to use as labels.

2. Your logs aren’t JSON formatted. Notably, it’s missing double quotes for the first level fields (for example, LEVEL: INFO is not json).

[imdkamrul1995]

@imdkamrul1995

level: “LEVEL”
class: “Class”
thread: “Thread”
ip: “IP”
host: “host”
message: “message”
requestParam: “requestParam”
requestBody: “requestBody”
requestHeader: “requestHeader”
responseHeader: “responseHeader”
responseBody: “responseBody”
responseStatus: “responseStatus”
requestId: “requestId”
source: “message”
drop_malformed: false

Thank you @b0b and @tonyswumac for your suggestion
Issue fixed.