Lost prometheus historic data while upgrading "kube-prometheus-stack" from From 21.x to 22.x through helm

  • What Grafana version and what operating system are you using?
    9.2

  • What are you trying to achieve?
    upgrading kube-prometheus-stack from 10.0.2 to latest version

  • How are you trying to achieve it?
    Upgrading step by step from kube-prometheus-stack 10.0.2 to 41.9.1 through helm chart using the official documentation provided by helm => kube-prometheus-stack 41.9.1 · prometheus/prometheus-community

  • What happened?
    During the upgrade process from the version 21.x to 22.x, there is an instruction to delete the kube-state-metrics
    [ kubectl delete deployments.apps -l app.kubernetes.io/instance=prometheus-operator,app.kubernetes.io/name=kube-state-metrics --cascade=orphan]

  • What did you expect to happen?
    I lost all my metrics data,even though PVC is mapped to prometheus
    post upgrade to 22.x, I am not able to see the history data.
    I am expecting all my historic data should retain post upgrade to latest version

  • Can you copy/paste the configuration(s) that you are having problems with?
    all the older data is lost

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Grafana dashboards view is also changed post upgrade

  • Did you follow any online instructions? If so, what is the URL?
    I followed the below URL for the upgrade process
    kube-prometheus-stack 41.9.1 · prometheus/prometheus-community


Now i am planning to perform the similar upgrade in my Production environment, but the historical data is very critical and cannot lose it. How can i upgrade without losing my historic data? otherwise did i missed anywhere to update any parameters in the values file?
Please advise, thank you in advance!!

Regards
Vamsi