Anomaly Detection in Grafana for Cloudwatch Datasource

Grafana Cloudwatch
1

  • What Grafana version and what operating system are you using?
    Grafana v10.1.5 (849c612fcb)

  • What are you trying to achieve?
    Create anomaly detection bands using Cloudwatch anomaly detection function for Cloudwatch metrics

  • How are you trying to achieve it?
    Tried in two different ways: Using Metrics Insights
    SELECT ANOMALY_DETECTION_BAND() from “”
    which seems to be limited to only AVG, COUNT, MAX, MIN, and SUM functions. It gives me metric validation error.
    Also tried with Expression->Math and calling ANOMALY_DETECTION_BAND($A).It gives me Query error 500

  • What happened?
    I get "metric request error: "ValidationError: Error in expression ‘query189c3d5142e44f8d9fc8aef2bc745cca’: Invalid syntax. ‘ANOMALY_DETECTION_BAND’ is not a valid aggregate function\n\tstatus code: 400, request id: 348c396c-911d-4552-99b7-c6ab86a59988\

  • What did you expect to happen?
    Expected to see two new time series for the upper and lower anomaly detection bands

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    "ValidationError: Error in expression ‘query189c3d5142e44f8d9fc8aef2bc745cca’: Invalid syntax. ‘ANOMALY_DETECTION_BAND’ is not a valid aggregate function\n\tstatus code: 400, request id: 348c396c-911d-4552-99b7-c6ab86a59988\

url:“api/ds/query?ds_type=expr&expression=true&requestId=Q167”

method:“POST”

data:Object

requestId:“Q167”

hideFromInspector:false

headers:Object

retry:0

traceId:undefined

message:"Query error: 500 "

2

Example how to use expressions - you have to use CloudWatch ID (e.g. m1 in my example), not Grafana ref ID (e.g. A) ($deviation is my dashboard numeric variable):

obrázok
obrázok1288×630 32.9 KB

3

Thanks for the quick response. I created a dashboard variable and tried using Cloudwatch ID as you suggested, however, I am still getting a validation error.

message:“metric request error: "ValidationError: Error in expression ‘queryb61cfeedeed649e388c170d02ec12357’: Unsupported operand type(s) for ANOMALY_DETECTION_BAND: ‘[Array[TimeSeries], Scalar]’\n\tstatus code: 400, request id: 50dbe5de-ffac-4e1c-87f3-3085507f3e3e"”

4 
Unsupported operand type(s) for ANOMALY_DETECTION_BAND: ‘[Array[TimeSeries], Scalar]

=> What is supported argument for ANOMALY_DETECTION_BAND?

ANOMALY_DETECTION_BAND - Arguments: TS, S (Timeseries, scalar)

So error is correct - array of timeseries (TS[]) is not supported

Don’t pass array (e.g. TargetResponseTime of multiple loadbalancers) - just pass/select single timeseries (e.g. single loadbalancer metric).

5

Thank you. Yes, the error was due to the array of timeseries as opposed to a single timeseries the function was expecting. So, now the error is gone, however I am not getting any data for anomaly detection function call. I am just retrieving a simple CPU Utilization metric from cloudwatch for a single host and trying to get anomaly detection to produce those bands. All I am getting now is the time series for the CPU utilization metric.

6

You must to enable Anomaly detection explictly in the AWS console for particular timeserie(s) first.

1 Like
7

Thank you so much! That solved my issue!

8

Thanks again for your help @jangaraj. I have a follow up question. I am trying to create a Grafana alert from this dashboard panel and I am not getting any data back for the ANOMALY_DETECTION_BAND queries from the alert panel. I get the regular Cloudwatch metric time series, but not the upper and lower time series. Any thoughts?

9

Hello grafmon24 and @jangaraj, I am in the same situation. The anomaly bands work in a dashboard, but when I try to create an alert, it shows as “No data”.

Did you find a way to fix it?

Thanks

10

Hello, Please see below.

Per the above link, the fix for the issue will be released in Grafana 10.2.0. Keep in mind that you will have to enable the ‘sseGroupByDatasource’ feature toggle for the time being, in order to use metric math in Alerts.

2 Likes