Lost prometheus historic data while upgrading "kube-prometheus-stack" from From 21.x to 22.x through helm

  • What Grafana version and what operating system are you using?
    9.2

  • What are you trying to achieve?
    upgrading kube-prometheus-stack from 10.0.2 to latest version

  • How are you trying to achieve it?
    Upgrading step by step from kube-prometheus-stack 10.0.2 to 41.9.1 through helm chart using the official documentation provided by helm => kube-prometheus-stack 41.9.1 · prometheus/prometheus-community

  • What happened?
    During the upgrade process from the version 21.x to 22.x, there is an instruction to delete the kube-state-metrics
    [ kubectl delete deployments.apps -l app.kubernetes.io/instance=prometheus-operator,app.kubernetes.io/name=kube-state-metrics --cascade=orphan]

  • What did you expect to happen?
    I lost all my metrics data,even though PVC is mapped to prometheus
    post upgrade to 22.x, I am not able to see the history data.
    I am expecting all my historic data should retain post upgrade to latest version

  • Can you copy/paste the configuration(s) that you are having problems with?
    all the older data is lost

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Grafana dashboards view is also changed post upgrade

  • Did you follow any online instructions? If so, what is the URL?
    I followed the below URL for the upgrade process
    kube-prometheus-stack 41.9.1 · prometheus/prometheus-community


Now i am planning to perform the similar upgrade in my Production environment, but the historical data is very critical and cannot lose it. How can i upgrade without losing my historic data? otherwise did i missed anywhere to update any parameters in the values file?
Please advise, thank you in advance!!

Regards
Vamsi

Hello Vamsi,

Have you by any chance somehow mitigated that.
I’m on the same issue here and cannot update a single config since I will loose all my data :frowning:

Thanks,
Stelios