Math expressions with multiple queries

  • What Grafana version and what operating system are you using?

Grafana v10.0.3 (eb8dd72637) on EKS cluster in multi-pod setup (2cores, 8gigs every pod)

  • What are you trying to achieve?

Building a traffic light dashboard that greps logs for status codes from different ElasticSearch / OpenSearch data sources and tries to combine them in one integer number representing if everything is good or bad (0 - all good, 1 - something is bad, 2 - something is critical)

  • How are you trying to achieve it?

I have around ~10 simple queries to different data-sources like on the following image

Output of each query is a simple two rows / two columns table like on the following image

Than I combine query results in simple math expressions like on the following image (test_a, test_b, and so on - references to queries). There is also a similar RED expression
image (16)

Than I have a result query defined as


This query outputs one normalised integer number (0 - all good, 1 - something is bad, 2 - something is critical) that than used to define the colour for visualisation.

  • What happened?

Without RED, ORANGE and OVERALL expressions overall queries execution time is less than 1s. With them being turned on - execution time goes to up 60 seconds and resource consumption on backend jumps to > 10 gigs of ram.

From the tests looks like OVERALL expression which uses results of other two expressions is the main problematic here.

And I have a list of following questions because of this behavior:

  • Are expressions evaluation lead to a queries being used being executed again on evaluation? Result of one query to datasource is just a simple table with not a lot of values and they are being processed in less than a second each of them, but with expression everything collapses
  • Expression evaluation is being done on backend, yes?
  • Am i using a wrong approach and something can be tweaked here?