This sounds like something between thanos and prometheus not being correctly configured. If I do the following query:
rate(process_cpu_seconds_total{instance=“$node”,job=“$job”}[$__rate_interval])
I get:
But if I remove the rate function:
process_cpu_seconds_total{instance=“$node”,job=“$job”}
I get:
So the data is clearly being stored, but the rate doesn’t get calculated for the data that is stored in thanos, vs the data that is stored in prometheus, which is the more recent 24h worth. The datastore is pointing at the thanos querier port. Any ideas?