- What Grafana version and what operating system are you using?
Grafana v10.0.3 (eb8dd72637) on EKS cluster in multi-pod setup (2cores, 8gigs every pod)
- What are you trying to achieve?
Building a traffic light dashboard that greps logs for status codes from different ElasticSearch / OpenSearch data sources and tries to combine them in one integer number representing if everything is good or bad (0 - all good, 1 - something is bad, 2 - something is critical)
- How are you trying to achieve it?
I have around ~10 simple queries to different data-sources like on the following image
Output of each query is a simple two rows / two columns table like on the following image
Than I combine query results in simple math expressions like on the following image (test_a, test_b, and so on - references to queries). There is also a similar RED expression
Than I have a result query defined as
2*$RED + $ORANGE
This query outputs one normalised integer number (0 - all good, 1 - something is bad, 2 - something is critical) that than used to define the colour for visualisation.
- What happened?
Without RED, ORANGE and OVERALL expressions overall queries execution time is less than 1s. With them being turned on - execution time goes to up 60 seconds and resource consumption on backend jumps to > 10 gigs of ram.
From the tests looks like OVERALL expression which uses results of other two expressions is the main problematic here.
And I have a list of following questions because of this behavior:
- Are expressions evaluation lead to a queries being used being executed again on evaluation? Result of one query to datasource is just a simple table with not a lot of values and they are being processed in less than a second each of them, but with expression everything collapses
- Expression evaluation is being done on backend, yes?
- Am i using a wrong approach and something can be tweaked here?