Bad Prometheus graphs starting with 4.6.2

4.6.1 on the left, 4.6.2 on the right

Could it be that this is due to "tsdb.HandleRequest() error invalid value type" · Issue #9777 · grafana/grafana · GitHub ? I believe I have the step interval matched. I’m working on updating Grafana in the Istio project. Our config is here:

https://github.com/istio/istio/tree/master/mixer/deploy/kube/conf
https://github.com/istio/istio/blob/master/install/kubernetes/addons/grafana.yaml

Prom config is here: https://github.com/istio/istio/blob/master/install/kubernetes/addons/prometheus.yaml

Thanks!

what is your scrape in interval setting? and what is your grafana step setting and rate scalar?

The data source scrape interval is 5s. Min step is ‘auto’. Not sure about rate scalar?
The dashboard json is here: https://github.com/istio/istio/blob/master/mixer/deploy/kube/conf/grafana-dashboard.json

Same dashboard json with 4.6.1 and 4.6.2. Only change is setting the scrape interval to match prometheus, as that setting is new.

I did some searching and am experimenting with the ‘Resolution.’ Changing it from 1/2 to 1/1 seems to make a big improvement.