Loki-distributed configuration for ingester persistence

Grafana Loki
1

So I’m trying to configure loki ingesters to use persistence for the data volume and am running into a strange problem.

In my values.yaml I have the following block under ingester:

    persistence:
      enabled: true
      claims:
        - name: data
          size: 20Gi
          storageClass: gp2

When I look at the volume clame of the ingester pod after applying the change I see the following:

  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: gp2

So far so good - thats what I believe it should look like. However when the pod tries to start up I get a failure:

Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    67s               default-scheduler  Successfully assigned loki/loki-app-dmor-loki-distributed-ingester-0 to ip-10-0-2-245.us-west-2.compute.internal
  Warning  FailedMount  3s (x8 over 67s)  kubelet            MountVolume.MountDevice failed for volume "pvc-54e2d7f6-336d-42d9-9c2a-f6cbde6fb0aa" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name efs.csi.aws.com not found in the list of registered CSI drivers

This should be using the gp2 ebs driver not the efs driver.

I looked over the template for the ingester - I don’t see that I’m missing anything. So I’m either looking something over or there is something in the ingester template that is missing maybe? Not sure where to go here - anyone have an idea?