Count over time returns no data for alert

I am trying to create an alert based on number of errors that are received over a time period. I am able to create a dashboard that easily counts the number of errors that occur over a time period.
Ex. there are 5 logs that are at the error level and the count shown on the dashboard is 5: count_over_time({application=“app_name”, level=“error”} [$__interval])

Putting this same query into alerts yields no data whatsoever. Any time I an able to get data back the count is wrong based on the time frame. I must be doing something wrong. Any help is appreciated!

First thing that comes up to my mind is that in alerting default format of Prometheus queries is Instant. Try to change it to Range