Hi, I’m a new user of grafana and I need to configure an alert with an elastic search datasource. The alert is for es search latency, the query is a combined one and is a derivative. As show below:
I need to alert when the search latency is over a certain threshold, say 5 seconds, the alert is as below:
But I can not get the right result, when I run the “Test Rule”, I got some output:
firing:false
state:""
conditionEvals:"false = false"
timeMs:"14.763ms"
logs:Array[2]
0:Object
message:"Condition[0]: Query Result"
data:Array[1]
1:Object
message:"Condition[0]: Eval: false, Metric: primaries.search.query_time, Value: null"
data:null
It seems the alert module cannot calc the final result of query A, it can just get some result from the first query which is a very large numeric number.
Can someone help me? With much appreciation !
PS: My grafana is 5.4.3
What does your graph panel look like? What time range are you using of the dashboard when testing the alert rule? Using Date Histogram with auto interval may be tricky in alerting situations where you usually looking at very short time range (last 5 min in your case).
The graph panel is as below:
I want to set an alert when the search latency is more than 3s(eg. 3000ms), sadly I cannot get the right result.
The time range of the dashboard is below:
Did I config something wrong or because grafana hasn’t supported this kind of alert yet?
Much appreciation for your reply.
I would change the time range in dashboard to last 5 minutes to simulate how the alerting query behaves and then include a screenshot and query inspector request/response result of that. Other things to try, change interval from auto in query to something else like 1 min.
Changing the interval of query to 1min solve the problem. If I set it to auto I found all data queried out are null values, when changed it to 1min I can get the right value. Thank you so much!
Maybe you haven’t configured a sane min interval for the datasource - A lower limit for the auto group by time interval. Recommended to be set to write frequency, for example 1m if your data is written every minute.
Does that mean I can set the interval of Metrics in a panel to auto(after set the interval of datasource to 1m)? I tried to do this but I still get a lot of(1500 for exactly) null in the alert, so I don’t quite understand.
Yes, that should work. You can set the min interval on datasource or per panel. Have you tried configuring on datasource?
