Hi,
I’m creating a company-wide dashboard for many of the metrics exposed by our microservice framework, and I’m seeing a strange behaviour on a heatmap. This dashboard includes variables that allow one to show multiple microservices at the same time, and the data I see on the heatmap seems different from what I would expect when multiple services are involved. Since they have partially different buckets, I’m wondering if that’s something I show allow at all.
To exemplify, these are the request time heatmaps on Service1 (top) and Service2 (bottom) separately:
Service1 higher buckets are 0.894784851, 0.984263336, 1.0 and +Inf
Service2 higher buckets are 0.894784851, 0.984263336, 1.073741824, 1.431655765, 1.789569706, 2.0 and +Inf
And the same heatmap for Service1+Service2:
As you can see, there are a few data points around the higher buckets that are not part of either individual heatmaps, precisely on the range of the buckets that are only present for Service2 (since it has longer request times and it was configured with a higher bucket limit). This is just a particular case where the buckets difference is very localized, but it can be much different in other services.
This is the query: sum(increase(http_server_requests_seconds_bucket{service=~"$service"}[$__interval])) by (le)
Can I fix the heatmap in some way so it shows appropriate data for multiple services, or is this kind of conciliation of different bucket ranges is simply not possible?