Permission denied accessing /var/log/pods in Grafana Alloy with hostPath volumes on Kubernetes

Grafana Alloy
1

I’m deploying Grafana Alloy on Kubernetes using a DaemonSet with hostPath volumes mounting /var/log/pods and /var/log/containers from the host. The pods have this configured in the spec:

  • The hostPath volumes have DirectoryOrCreate type set correctly.
  • The mounted directory /var/log/pods on the host has permissions drwxr-xr-x root:root.

However, inside the container, I get Permission denied when listing /var/log/pods. I have tried running as root user and group, but it still fails.

Also getting an error “source.file.pods error="stat /var/log/pods///.log: no such file or directory” filename=/var/log/pods///.log’.

What best practices should I follow to fix permission issues with hostPath mounts for log directories like /var/log/pods in Grafana Alloy? Should I use init containers to fix permissions, or adjust fsGroup differently? How can I safely allow Alloy to read these log directories without compromising cluster security?


apiVersion: collectors.grafana.com/v1alpha1
kind: Alloy
metadata:
  name: alloy-logs
  namespace: test
spec:
  controller:
    type: daemonset
    # enableStatefulSetAutoDeletePVC: true
    tolerations:
      - effect: NoSchedule
        operator: Exists
    volumes:
      extra:
        - name: pod-logs
          hostPath:
            path: /var/log/pods
            type: DirectoryOrCreate
        - name: container-logs
          hostPath:
            path: /var/log/containers
            type: DirectoryOrCreate            
        - name: data-vol
	  hostPath:
            path: /var/alloy
            type: DirectoryOrCreate
        - name: alloy-client-cert
          secret:
            secretName: alloy-client-cert
    volumeMounts:
      extra:
        - name: pod-logs
          mountPath: /var/log/pods
          readOnly: true
        - name: container-logs
          mountPath: /var/log/containers
          readOnly: true
        - name: alloy-client-cert
          mountPath: /etc/alloy/certs
          readOnly: true
  serviceAccount:
    create: false
    name: alloy-sa
    automountServiceAccountToken: true
  rbac:
    create: true
  alloy:
    mounts:
      varlog: true
      extra:
      - name: data-vol
        mountPath: /var/alloy
        readOnly: false
    configMap:
      create: false
      name: alloy-config
      key: alloy-logs
    storagePath: /var/alloy
    securityContext:
        runAsUser: 0
        runAsGroup: 0
        fsGroup: 0

configmap:

apiVersion: v1
data:
alloy-logs: |
logging {
level = “debug”
}

loki.source.file "pods" {
  targets = [
    { __path__ = "/var/log/pods/*/*/*.log" },
  ]
  forward_to = [loki.process.pod_logs.receiver]
}

loki.process "pod_logs" {
  forward_to = [loki.write.victorialogs.receiver]
}

loki.write "victorialogs" {
  endpoint {
    url        = "endpoitnt"
    tenant_id  = "0:0"
  }
  external_labels = {}
}

kind: ConfigMap
metadata:
name: alloy-config