Invalid extended statistics - Hosted Grafana + Cloudwatch metrics broken overnight

We came into work today and many of our Grafana charts are now showing the error “Invalid extended statistics” instead of data.

We’re on grafana.net and get the metrics from AWS CloudWatch metrics. It looks like most of them are pulling data from AWS/ApplicationELB, for example “HealthyHostCount”.

Any ideas how to debug this?

Found the issue. It’s related to https://github.com/grafana/grafana/issues/13633.

We have heaps of metrics reading p95. Apparently now they need to say “p95.00”.

Ridiculous. What a stupid breaking change.