CloudWatch with Multi-Value Template Variable and Math expression

Is it possible to use CloudWatch math expressions together with multi-value template variables?

For instance see screenshot: I have a template variable that is a list of all Firehose names and I want to query a particular metric that requires post-processing (normalization to rate/second) in CloudWatch math expressions. When using the template variable in a plain CloudWatch query, everything is fine and Grafana shows n graphs, where n is the number of names in the variable.

However, when i give the query an Id and use it in another query, everything disappears and Grafana shows “no data points” - and no error message.

Judging from the query inspector, Grafana seems to exten the Id with some random strong for every of the n sub queries, like id:"rawc2VnbWVudC1pbXBvcnQ0".

This is how we do a similar thing with classic loadbalancers.
A template variable contains all the elb names.
The first query (which is hidden) gets the HealthHostCount using the variable which returns multiple series.
Then the second query is a math query on the first, which returns a single series.

Thank you very much! This wasn’t the complete solution to my problem, but let me realize a few things:

  • one can reference the metrics from A in B (in your example) via METRICS(), although the specific IDs are, as I mentioned, randomized

  • however, this didn’t help me because i want the rate that I needs id / PERIOD(id) (can’t use the RATE() function because it requires an ever increasing counter, not a gauge like CloudWatch stores…)

  • but I realized that one can skip using Grafana’s UI query builder completly by using the SEARCH() function in Expression from the beginning and that {{ label }} as Alias shows the values of dimension names one gives as parameter to SEARCH:

    SEARCH(' {AWS/Firehose,DeliveryStreamName} MetricName="DeliveryToS3.Records" ', 'Sum', 300) / 300

    The period can probably be improved with $__interval_ms or such, but basically that’s how it can work

  • using a single query and not one hidden and one Expressioned one also works around this regression in 6.2

This is all not really documented as far as I understood, which is a shame. Maybe I will create a merge request for improving the docs, but so long I hope that this thread might help others.

2 Likes

Glad it helped a bit.
Thanks for replying, I wasn’t aware of the search expressions.
I’ll have to look into them more as it may help me with something I’m struggling with.

Hello Sebastian, Hope you are doing well, I am trying create alert in grafana for CloudWatch metrics, but I couldn’t because it not supported when I am referencing metric from another query,
how did you setup several metrics of CloudWatch in 1 query?

@barmolei My last response (CloudWatch with Multi-Value Template Variable and Math expression - #3 by sebastianbrandt) is still my latest knowledge :slight_smile: