Loki is not able to fetch fields from the log line

Hello Everyone,

This is my log json:

{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal",
"short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}",
"full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}",
"timestamp":1.707740449767E9,
"level":6,
"facility":"logstash-abctest",
"simpleLogger":"test-abc",
"logSequence":9696969696,
"activity":"00000000222255544",
"logLevel":"INFO",
"logger":"test",
"senderType":"test",
"terminalID":"test300"}

My application is running on ECS fargate and I have added logConfiguration in my running tasks as below:

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "LabelKeys": "activity,full_message,senderType,container_name,ecs_task_definition,source,ecs_cluster",
                    "Labels": "{job=\"firelens\"}",
                    "LineFormat": "json",
                    "Name": "grafana-loki",
                    "RemoveKeys": "container_id,ecs_task_arn",
                    "Url": "http://loki.endpoint:3100/loki/api/v1/push"
                }

My question is: I have added log labels in the ‘LabelsKeys’ but I am only seeing limited lebels in Grafana UI Explorer, check Screenshot.

Thanks in advance for your valuable feedback.

  1. Can you show what your logs actually look like in Grafana?

  2. I haven’t used firelens in quite some time. I would recommend logging into the firelens container, grab the generated fluentbit configuration, and then you can test the logic easily on your workstation.

@tonyswumac Thanks for your reply. Here is the actual log line:

{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal","short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}","full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}","timestamp":1.707740449767E9,"level":6,"facility":"logstash-abctest","simpleLogger":"test-abc","logSequence":9696969696,"activity":"00000000222255544","logLevel":"INFO","logger":"test","senderType":"test","terminalID":"test300"}

Looking at firelens documentation, it would seem that you need to specify a config file for parsing JSON (see Example logging option task definitions - Amazon Elastic Container Service). Unfortunately I can’t of much help, since I haven’t used it for a long time. I’d still recommend grabbing the generated configuration from the container and just poke at it and see exactly what it’s doing.

Hi @tonyswumac … Thanks again for checking. My actual requirement is to filter data from the logs fields. Let me give you a breif intro. I recently changed logging from AWS cloudwatch to Grafana Loki. Team on Cloudwatch getting data from the query in Cloudwatch log insights. Query is below:

fields *@timestamp* , activity, terminalID, clientID, full_message
| filter full_message like “Configuration”
| filter clientID like “121212121121"
| sort *@timestamp* desc
| limit 2000

My question is: How can I run the similar query in Loki (LogQL).? if you can help please.
Thank you in advance.

I would recommend you to read through LogQL: Log query language | Grafana Loki documentation.

To parse JSON, you would simply do {SELECTOR} | json, see LogQL Analyzer | Grafana Loki documentation