Configure local filesystem storage for Loki

Grafana Loki
1

I’m running Grafana and Prometheus successfully on my microk8s cluster. I’m now trying to setup Loki. It’s mostly working, however I’m still unable to configure the local filesystem. I’m using the loki-stack helm charts to deploy.

Here is what I did:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki-stack -f loki-values.yaml

And here is the content of loki-values.yaml:

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  commonStorageConfig:
    type: filesystem
  rulerConfig:
    storage:
      type: local
  storage_config:
    filesystem:
      directory: /home/djipey/loki
singleBinary:
  replicas: 1
  persistence:
    enableStatefulSetAutoDeletePVC: false
    storageClass: local-loki
    enabled: true
    size: 5Gi
promtail:
  config:
    snippets:
      pipelineStages:
      - docker: {}
      - cri: {}
      - json:
          expressions:
            level: level
            path: path
            ts: ts
            time: time
            status: status
            method: method
            latency: latency
      - labels:
          level: ''
          path: ''
          status: ''
          method: ''
      - timestamp:
          source: ts
          format: RFC3339Nano
  enabled: true

For loki’s PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki-db
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /home/djipey/loki
    type: DirectoryOrCreate
  storageClassName: local-loki

The dir /home/djipey/loki exists on the machine and I chmoded it 777 so I don’t expect any permission issues.

The setup above works, and I can create a Loki data source in Grafana, and it contains the data I expect. However, the loki folder stays empty. A describe on the loki pod hints that the storage config is ignored:

Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  loki
    Optional:    false
  storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-w5tz2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

But it doesn’t work, the PV is never claimed and the loki folder on my machine stays empty. I have been at it for 2 days now and I can’t get it to work.

Here are the versions I’m using:

❯ helm list
NAME                            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                   APP VERSION
kube-prometheus-stack           monitoring      14              2023-05-18 00:06:08.505163787 +0100 BST deployed        kube-prometheus-stack-45.28.1           v0.65.1    
loki                            monitoring      8               2023-05-19 11:55:18.176228629 +0100 BST deployed        loki-stack-2.9.10                       v2.6.1     
prometheus-postgres-exporter    monitoring      3               2023-05-17 23:36:51.728690574 +0100 BST deployed        prometheus-postgres-exporter-4.4.3      0.11.1

Would you have any idea how to configure that please?

1 Like