I am monitoring a JVM / Kamon based system, and one of the metrics is looking at time spent serving requests.
I run a performance test and get a good graph out of how many requests are happing over time
BUT the second graph is the histogram that should give the 99 percentile of these requests, but here the histogram does not go down as the requests go down, it remains… why ?
and how can I have a histogram that follows, or otherwise I suspect I will get old data polluting my stats. Or am I perhaps misunderstanding how it should be interpreted.
I mean 17.20 the P99 cannot be some 600 ms, cause there are no requests…
Some details:
Metrics: max by(parentOperation,operation) (histogram_quantile( 0.99,span_processing_time_seconds_bucket{job=~“$job”, http_method=“GET”}))
Legend: {{parentOperation}} x {{operation}} P99 {{http_status_code}} {{error}}
