How to show a service is down when a time series ends?

Grafana
,
1

Using version v8.3.3 (Grafana Cloud)

Im trying to make use the State timeline to represent if a few of our services are up or not. I am trying to do this by checking if a counter keeps increasing for these services with a query such as:

count((rate(a_counter_total{select=~ ".+"}[$__interval]))) by (select)

The problem seems to be that when the counter doesnt get updated anymore with new values, the state timeline hangs on the last value and will thus forever show up as green. Even though i expect (and it did seem to work that way until yesterday) that the state timeline would be able to show a different colour for this “no value”.

State timeline:
Treshholds set so that base is red, anything > 0.5 is green and everying > 1.5 is blue

afbeelding
afbeelding1889×534 8.3 KB

When looking at a time series for one of these services you can see that the series ends. and as such i would expect the state timeline to fall into the red category. Yet on the timeline nothing is visible.

afbeelding
afbeelding2113×469 14.5 KB

Maybe im just going about this wrong ofcourse, any help would be appreciated. Though id like to note that this seemed to work up untill yesterday.

2

Hi @yannickbus and welcome to the community!

If you want input on other potential queries to evaluate the state of the service, can you also share an example of the source data you’re using to create the counter? If you can share your raw unformatted data, it can be useful for the community to mock up your use case to offer suggestions:

I suspect that because this is a counter it will always have an increasing or repeating value, so that might be why it’s not meeting a different threshold condition.

The value mapping option has the ability to change colors based on special conditions like null values. Knowing what your data looks like will be helpful in suggesting an approach for your use case.