How can i expand a single log entry from Loki into multiple rows

  • What Grafana version and what operating system are you using?
    Grafana v10.1.1 (0cfa76b22d) running in a K8s cluster. Client machine is latest MacOS running Chrome

  • What are you trying to achieve?
    I have retrieved a single logline from Loki using limit=1. That logline contains a json object, including an array.
    I want to use the Table panel to show a line for each element in the JSON array

  • How are you trying to achieve it?
    I have tried various transformations, including the Extract field transformation, but i’m still left with just one row in my table.

  • What happened?
    I can’t get to a point, where i can show a row for each item in the JSON array

  • What did you expect to happen?
    I was expecting to be able to pass a JSON array to the Table panel, and it would then iterate through the items, generating one row per array entry.

  • Can you copy/paste the configuration(s) that you are having problems with?
    My logline holds this data

{
  "snapshot": [
    {
      "title": "Microsoft Azure Information Protection Unified Labeling Client v2.16.79.0",
      "update_revision": "200"
    },
    {
      "title": "2023-10 Update for Windows 10 Version 21H2 for x64-based Systems (KB4023057)",
      "update_revision": "1"
    },
    {
      "title": "Feature update to Windows 10, version 22H2",
      "update_revision": "1"
    },
    {
      "title": "2023-10 Cumulative Update for .NET Framework 3.5, 4.8 and 4.8.1 for Windows 10 Version 21H2 for x64 (KB5031223)",
      "update_revision": "200"
    }
  ]
}

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors. I just have one row in the table which contains all the data.

  • Did you follow any online instructions? If so, what is the URL?
    Transform data | Grafana documentation

1 Like

I want to achieve this too… So sad, no help. :frowning:

May be this comes a little bit late, but I’ll drop it here in case it’s useful for somebody:

We are using Loki as datasource, extracting the json and then doing some math. We managed to get one table row for each serie returned using a transformation called Reduce, and mode Series to Rows.

Not sure our series are like yours, but hope it helps

@rgg1993 Thanks for sharing.
Can you share what your JSON logline looks like (the general structure), and how you query it out?

Sure.

Our objective was to have measured the response time grouped by endpoint. So we did the following:

  1. We are getting logs only from a specific environment and job, filtering those that don’t have request completed and the selected endpoint.

{environment="$environment",job="$my-job-name"} |~"request completed" |~"$endpoint"

Output:

{
“container_name”: “/my-container”,
“environment”: “demo”,
“hostname”: “my-hostname”,
“job”: “my-job-name”
}

  1. With the json transformation this is is expanded and brings the field inside the log field and retrieve the value for the label endpoint and filter matches

{environment="$environment",job="$my-job-name"} |~"request completed" |~"$endpoint" | json | label_format endpoint="{{.log_payload_endpoint_path}}" | endpoint=~"$endpoint"

Sample output (note that endpoint will be listed in the label endpoint and also in the original field name log_payload_endpoint_path)


{
  "container_id": "123456",
  "container_name": "/my-container",
  "endpoint": "/",
  "environment": "environment",
  "job": "my-job-name",
  "log_msg": "request completed",
  "log_payload_endpoint_path": "/my-chosen-endpoint",
  "log_payload_res_statusCode": "200",
  "log_payload_responseTime": "1",
  "log_time": "1717070875817",
  "source": "stdout"
}

This query output is still shown as a table, with one line per timestamp:

  1. when we add the avg_over_time and the unwrap the query is as follows:

avg_over_time({environment="$environment",job="$my-job-name"} |~"request completed" |~"$endpoint" | json | label_format endpoint="{{.log_payload_endpoint_path}}" | endpoint=~"$endpoint" | unwrap log_payload_responseTime [$__interval]) by (endpoint)

And we get one serie for each endpoint. In time series format it shows like one coloured dot for each endpoint:

In a table format it’s one table for each serie:

That’s when we add the transformation to reduce:

It will require to choose table view, and then you should get something like this:

And that’s it.

Hope this helps you out :sunflower:

1 Like