Loki not parsing/detecting fields in logs

[michaelrichter1512]

Hi there,

I’m currently trying to properly utilize Loki for my logs. My services are running in kubernetes and the logs are being scraped by Grafana Agent.

This is an example json log from my service:

{"message":"SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo for db-server.mysql.database.azure.com failed", 
"level":40, "level_name":"ERROR",
"channel":"testing",
"datetime":"2023-08-07T06:08:17.641956+00:00",
"extra":{}}

However, in Grafana (Loki Explorer) if I try to query them with “{service} | json” it throws an error with “Value looks like object, but can’t find closing ‘}’ symbol”. The curly brackets are correct but the logs themselves start like that in the UI:

2023-08-07T06:26:57.76925542Z stderr F {"message"...

and if I extend the log the only detected fields are kubernetes related ones, not from the log/json itself.

How can I fix this?

Any help is appreciated!

[b0b]

What does your Grafana Agent config look like?

From what I can see I suspect you are doing

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - docker: {}

But the log format suggest that ContainerD is used on the nodes and you should be using cri instead of docker like

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}

That might be all you need to change.

[michaelrichter1512]

That was actually it, can’t believe it was the mistake. Thanks a lot!