For few days now I’m trying to create alert when ratio produced to consumed messages is not equal to 1. I don’t want to do it based on prometheus due to each time service is restarted, counter is reseted also I don’t want to create gauge because I’d have to store count of produced messages somewhere in db. I decided to use logs stored in elasticsearch. Here the problem is that service which produces messages is logging to one elasticsearch and service which consumes them to another elastic. According to my knowledge it is impossible to create alret based on 2 queries.
I’m out of ideas how to approach this in other way around. Maybe I’m missing something or there is other way how to create alert like this based on 2 data sources?