Best way to filter data from Elasticsearch

Note: Due to my company’s policy I can not post an specific details about the data, just broad ones, so any exact names for things have been removed or changed.

I am using Grafana v12.2.1 running inside a docker container in an Ubuntu VM on a server. Elasticsearch is running on a different VM on another server.

My data has the structure:

  • lvl1
    • lvl2 (more than 1)
      • lvl3 (more than 1)

When I uploaded it to Elasticsearch I inverted it. So in Elasticsearch, each document is 1 lvl3 which includes the information about the lvl2 and lvl 1 it came from. So it looks like:

  • lvl3
    • data
    • lvl2 (the parent)
      • data
      • lvl1 (the “grandparent”)
        • data

Obviously anytime something comes from the same parent or grandparent that part of the data will be identical over all of the documents.

In lvl1 I have some data that is an array of key:value pairs. So it looks like:
name:[key1:value1, key2:value2, ….]

Inside of elastic search I am able to search using name.keyword:key1* and it gives me every document that has key1 in it (not all document have it).

In Grafana, I want to get a count for each wildcard match and make a chart with it. So if for example that key has 3 different values it can be, I want to know how many documents have each value.

When I try to do a query with a filter of name.keyword:key1* and then group by name.keyword I get “Post “https://IP:9200/\_msearch?max_concurrent_shard_requests=5”: net/http: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)”

What is the best way to do this? Please let me know if anything is unclear

Can you at least post what the data structure in json looks like? I assume the data is a json document?

It’s a JSON file, yes. It would be too complicated to post the whole data structure. I just checked the data structure, it is mostly flat. At least for the parts that matter for what I am trying to do, it is all flat. So everything looks like:
{

id:value

somedate:value

somelist:[key1:value1, key2:value2, …]

.

.

.

}

a somelistlooks like:

“somelist”: {
“properties”: {
“id”: {
“type”: “text”,
“fields”: {
“keyword”: {
“type”: “keyword”,
“ignore_above”: 256
}
}
},
“somevalue”: {
“type”: “text”,
“fields”: {
“keyword”: {
“type”: “keyword”,
“ignore_above”: 256
}
}
}
}
}

So I am trying to get something out of a somelist

I hope this helps enough.

1 Like

somelist:[key1:value1, key2:value2, …] but some list in the json does not look like an array?

It is an array in the json. And using the .keyword allows you to search inside of an array. So I can do something like somelist.keyword:key1* or somelist.keyword:key2*. I didn’t create the json originally, but I think that the key:value pairs aren’t actual json in there just plain text

i would go with infinity it has some powerful features.

1 Like

Could you give me a bit more information about to to use Infinity with ElasticSearch? I did some quick googling and wasn’t able to find much