Parsing JSON list of objects using transformations with Elasticsearch data source

Grafana Dashboards
1

I’m using Grafana version 10.1.6. I’m setting up metricbeat to monitor Elasticsearch/Logstash. In the logstash stats, I’m receiving an array of JSON objects (without any keys) for one of the nested fields. Since it is an array, I’m not directly able to graph it on a panel, and I have tried to parse it using transformations.

Is there any straightforward way to parse the array? The extract fields transformations using JSON format isn’t working, since I would need a way to index the array elements. I tried the JSON array into a string and then extracting the key-value pairs, but that does not seem to catch all the required fields.

Thanks!

2

we are not able to take a peek at your pc remotely so please provide sample data as inline text here

3

Sorry! It is something like this:

{
  "field_1": [
    {
      "inner_field_1":  "value1"
      "inner_field_2": "value2"
    },
    {
      "inner_field_3":  "value3"
      "inner_field_4": "value4"
    },
    {
      "inner_field_5":  "value5"
      "inner_field_6": "value6"
    }
  ]
}

I was able to simplify it a bit. I used the transformation “extract fields” on “field_1” using key-value format. Now I have 3 columns (0, 1, 2) with JSON objects. Is there a way I could do something like a union of the columns (with more rows basically)?

1 Like
4

ao if you have field_1 should we assume you have field_n+1 fields?

5

Its just that one field that I want to extract data from. I mean there are other (non-nested) fields that metricbeat is sending over which are being parsed normally by grafana (as in, I’m able to view the statistic by just apply a metric over the field).

1 Like
6

gotcha. what if you add yet another extract transformation to extract the extracted json array?

7

Yep, that was the plan once I’ve gotten all the columns combined under one column. So right now I have 3 columns, each contain JSON objects with the same structure. If I’m able to combine these columns then I’d be able to extract the data and apply filters as required.

8

I figured out a solution, I parsed each column individually then used the add binary operation on the field that I wanted to aggregate it to one column. Probably not the most clean solution but it works. Thanks for your help @yosiasz !

1 Like
10

great job! this is why I use infinity plugin with elastic search using url:9200 and use jsonata to do all kinds of cool stuff

1 Like