Loki is not able to fetch fields from the log line

Grafana Loki
1

Hello Everyone,

This is my log json:

{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal",
"short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}",
"full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}",
"timestamp":1.707740449767E9,
"level":6,
"facility":"logstash-abctest",
"simpleLogger":"test-abc",
"logSequence":9696969696,
"activity":"00000000222255544",
"logLevel":"INFO",
"logger":"test",
"senderType":"test",
"terminalID":"test300"}

My application is running on ECS fargate and I have added logConfiguration in my running tasks as below:

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "LabelKeys": "activity,full_message,senderType,container_name,ecs_task_definition,source,ecs_cluster",
                    "Labels": "{job=\"firelens\"}",
                    "LineFormat": "json",
                    "Name": "grafana-loki",
                    "RemoveKeys": "container_id,ecs_task_arn",
                    "Url": "http://loki.endpoint:3100/loki/api/v1/push"
                }

My question is: I have added log labels in the ‘LabelsKeys’ but I am only seeing limited lebels in Grafana UI Explorer, check Screenshot.

Thanks in advance for your valuable feedback.

Screenshot 2024-02-12 at 13.31.44
Screenshot 2024-02-12 at 13.31.44259×513 19.3 KB

2

  1. Can you show what your logs actually look like in Grafana?

  2. I haven’t used firelens in quite some time. I would recommend logging into the firelens container, grab the generated fluentbit configuration, and then you can test the logic easily on your workstation.

3

@tonyswumac Thanks for your reply. Here is the actual log line:

{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal","short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}","full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}","timestamp":1.707740449767E9,"level":6,"facility":"logstash-abctest","simpleLogger":"test-abc","logSequence":9696969696,"activity":"00000000222255544","logLevel":"INFO","logger":"test","senderType":"test","terminalID":"test300"}

4

Looking at firelens documentation, it would seem that you need to specify a config file for parsing JSON (see Example logging option task definitions - Amazon Elastic Container Service). Unfortunately I can’t of much help, since I haven’t used it for a long time. I’d still recommend grabbing the generated configuration from the container and just poke at it and see exactly what it’s doing.

5

Hi @tonyswumac … Thanks again for checking. My actual requirement is to filter data from the logs fields. Let me give you a breif intro. I recently changed logging from AWS cloudwatch to Grafana Loki. Team on Cloudwatch getting data from the query in Cloudwatch log insights. Query is below:

fields *@timestamp* , activity, terminalID, clientID, full_message
| filter full_message like “Configuration”
| filter clientID like “121212121121"
| sort *@timestamp* desc
| limit 2000

My question is: How can I run the similar query in Loki (LogQL).? if you can help please.
Thank you in advance.

6

I would recommend you to read through LogQL: Log query language | Grafana Loki documentation.

To parse JSON, you would simply do {SELECTOR} | json, see LogQL Analyzer | Grafana Loki documentation

7

Hi @tonyswumac . This issue still not resolved. I did not find anything in firelense. Can you recommend any other log log collector which can parse json fields as labels.

Thanks in advance.

8

Can you start from the top, please?

You have some logs in Loki that are JSON format, yes? What happens when you use the json filter on them? Can you share a screenshot?

9

@tonyswumac Let me rephrase the issue:

My application running on ECS cluster. I have grafana, loki on the same ECS cluster. My application log line is below:

{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal","short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}","full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}","timestamp":1.707740449767E9,"level":6,"facility":"logstash-abctest","simpleLogger":"test-abc","logSequence":9696969696,"activity":"00000000222255544","logLevel":"INFO","logger":"test","senderType":"test","terminalID":"test300"}

Log has some fields like host, short_message, full_message, facility, sendType and so on.

Here is my logConfiguration from task defination:

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "LabelKeys": "$host,$short_message,$full_message,$short_message,$senderType",
                    "Labels": "{job=\"c2-pmsrep\"}",
                    "LineFormat": "json",
                    "Name": "grafana-loki",
                    "RemoveKeys": "container_id,ecs_task_arn",
                    "Url": "http://loki.testenv:3100/loki/api/v1/push"
                }
            },

In Grafana, I want to filter data using Labels like full_message, sendterType, host etc. But I am not able to see Json fields (host, short_message, full_message, facility, sendType) as Labels.

I don’t know what are the necessary configuration required to achieve this.

Thank you so much in advance. You can see Grafana UI Screenshot:

image

10

As I mentioned, send your logs through the JSON filter. If you are not sure how to do that, please see LogQL: Log query language | Grafana Loki documentation.

11

Hi @tonyswumac Thanks for your update. But I am not able to understand. The link you have provided is to use filters, expression or operators of LogQL. We can use in order to query data Loki.

My question is simple. I am not able to parse my json fields (full_message, short_message, timestamp etc) and ship to Loki. (you can take a look at the sample log strem above)

I am using AWS fluendbit + firelense to collect, parse and ship logs to Loki data source. I am sure I need to make some changes into firelense configuration

"LabelKeys": "$host,$short_message,$full_message,$short_message,$senderType",
                    "Labels": "{job=\"c2-pmsrep\"}",
                    "LineFormat": "json",

But not sure about the syntex in ECS Task defination. Someone has already done it with the below configurations:

labels      app=${SVC_NAME}, version=${SVC_VERSION}, env=${ENV}
    label_keys  $ecs_cluster, $ecs_task_definition, $container_id

I am only able see meta data of cluster (container_name, ecs_cluster, ecs_task_definition, job, source) as Lebels.

But I need to make necessary configuration to in ECS fargate Task defination in order to parse and ship json fields as Lebels.

I hope I can make you understand my requirement.
Thanks in advance.

12

Here is a simple example:

  1. Raw json log:

Screenshot 2024-03-15 at 11.32.25 AM
Screenshot 2024-03-15 at 11.32.25 AM1960×1040 163 KB

  1. Json log parsed with json filter:

Screenshot 2024-03-15 at 11.32.44 AM
Screenshot 2024-03-15 at 11.32.44 AM1954×1152 192 KB

13

Have you been able to fix it? I think i have the same issue. I am able to push static labels to loki, but no luck with extracting labels out of a log record.

Trying to find a way to use “labelKeys” as mentioned in the fluentbit configuration, but not luck. In my scenario, i would even need to extract nested labels (in worst case, i a can still change my logs). I tried as well the access pattern with $, but not luck. My last restort might be, writing a LabelMapPath JSON file. Haven’t tried that and i would like to avoid that.

14

Hi @Django …
I have tried all possible ways but no output. And I also found out that Loki has limited search capabilities. Even if you able to get lables, you cannot search a particular string from the label. For example, I have a Field and data below:

full_message":"Apply existing Activity ID from header

if I search for string ‘existing’ in labels full_message, I got empty output. Therefore I am not sure Loki has sufficient capabilities when it comes to search…

I am switching to Elasticsearch… thats the last resort for me… Sadly.