Can't run Helm upgrade when using PVC as a persistence.type

Grafana
, ,
1

Hi all,

I recently incorporated Grafana into our k8s stack. I liked that I was able to deploy via a Helm chart and customize my deployment (load dashboards, configure ingress, set up SSO, etc etc) with the values.yaml. This is working great, and being deployed by an ADO pipeline to my production environment without issue on the first time it deploys. The thought process is that we’d iteratively upgrade the Grafana deployment via values.yaml (add dashboards, etc) and then just push that out via that same pipeline. The issue I am running into is that I cannot run helm upgrade without it just timing out. The PVC that has been created just gets stuck at a terminating state for 5 minutes and then the deployment fails. If I helm delete the deployment, and start fresh, it works as intended. I would like to be able to run helm upgrade without issue though.

I have tried switching everything around from all of the documentation I have seen, but I get the same results every-time.

  • Tried readwriteonce and readwritemany
  • Tried specifying an existing claim
  • Tried commenting out the finalizer (as it looked like that could maybe lock the pvc)
  • Tried different storage classes
  • Tried deploymentStrategy.type RollingUpdate & Recreate

I thought maybe it needed to be statefulset instead of pvc, but when I switch the type, it seems that the rest of my provisioning gets busted - so I quickly backpedaled away from that as an option.

I would love some help, or at least a guiding question that will send me to the next area of troubleshooting.

This is what I am trying to accomplish

  1. Deploy Grafana via Helm chart (release 1)
  2. Make changes to our Grafana requirements, implement those in values.yaml
  3. Run a Helm upgrade on the Grafana release via an ADO pipeline
  4. The chart would be upgraded with the changes referenced in the values.yaml. PVC recreated and pipeline executing without error.
  5. All good.

How it is currently working

  1. Deploy Grafana via Helm chart (release 1)
  2. Make changes to our Grafana requirements, implement those in values.yaml
  3. Upgrade times out, PVC gets stuck in terminating state.
  4. I have to delete the Grafana Helm release from my k8s cluster
  5. Re-deploy those changes made in step 2 as a fresh Helm install.
  6. All good. Can’t run Helm upgrade when using PVC as a persistence.type

Here is my persistence configuration currently:
image

Storage class that I am using:

image
image1083×237 24.5 KB

1 Like