Alloy DaemonSet & Log Position File / State Persistence

Grafana Alloy
1

Hi all,

I’m setting up Grafana Alloy as a DaemonSet to collect pod logs from each node. I understand that Alloy keeps a position file or state file to track which log lines it has already read.

My question:

  • If an Alloy pod restarts on a node, will it re-read logs from the beginning, or does it persist the position somewhere?
  • What’s the recommended approach to avoid duplicate log ingestion in this scenario?
    • Should I mount a hostPath volume for Alloy’s storagePath?
    • Or is there a built-in way for Alloy to persist state per node?

I want to make sure we don’t duplicate log lines while still using DaemonSet mode for full node coverage.

2

You should do this.

3

For persisting Alloy’s position/state file in DaemonSet mode, is this the correct way to configure it in the Alloy operator values file?

alloy:
  storagePath: /var/lib/alloy/data
  mounts:
    extra:
      - name: data-vol
        mountPath: /var/lib/alloy/data

And then define the volume:

controller:
  volumes:
    extra:
      - name: data-vol
        hostPath:
          path: /var/lib/alloy/data
          type: DirectoryOrCreate

I want to confirm that this ensures Alloy keeps its state across pod restarts without duplicating logs.

4

Yes, that looks correct. You’ll want to test it a bit, of course.

5

Error: failed to create the remotecfg service: mkdir /var/lib/alloy/data/remotecfg: permission denied

6

Make sure you mount the directory with RW, and that it’s writable to all on the host if you aren’t running alloy in privileged mode.

7 
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 0
8

Is the mount RW?

9

i tried this way.

extra:
- name: data-vol
mountPath: /var/lib/alloy/data
readOnly: false # ensure writable

10

Since you mentioned daemon set, I assume you are using kubernetes? Can you post your entire helm chart or whatever your deployment uses?

11 
apiVersion: collectors.grafana.com/v1alpha1
kind: Alloy
metadata:
  name: alloy-logs
  namespace: test
spec:
  controller:
    type: daemonset
    # enableStatefulSetAutoDeletePVC: true
    tolerations:
      - effect: NoSchedule
        operator: Exists
    volumes:
      extra:
        - name: pod-logs
          hostPath:
            path: /var/log/pods
            type: DirectoryOrCreate
        - name: container-logs
          hostPath:
            path: /var/log/containers
            type: DirectoryOrCreate            
        - name: data-vol
	  hostPath:
            path: /var/alloy
            type: DirectoryOrCreate
        - name: alloy-client-cert
          secret:
            secretName: alloy-client-cert
    volumeMounts:
      extra:
        - name: pod-logs
          mountPath: /var/log/pods
          readOnly: true
        - name: container-logs
          mountPath: /var/log/containers
          readOnly: true
        - name: alloy-client-cert
          mountPath: /etc/alloy/certs
          readOnly: true
  serviceAccount:
    create: false
    name: alloy-sa
    automountServiceAccountToken: true
  rbac:
    create: true
  alloy:
    mounts:
      varlog: true
      extra:
      - name: data-vol
        mountPath: /var/alloy
        readOnly: false
    configMap:
      create: false
      name: alloy-config
      key: alloy-logs
    storagePath: /var/alloy
    securityContext:
        runAsUser: 0
        runAsGroup: 0
        fsGroup: 0
12

Are you sure that’s correct?

13

image
image715×165 14.8 KB

but volumeMounts is not avilable.

reference : https://raw.githubusercontent.com/grafana/alloy/main/operations/helm/charts/alloy/values.yaml

14

I don’t use that helm chart myself, maybe your configuration is right. I would double check your container, check your pod description, and figure out why the mount is not writable.

15

yeh i will check that

16

Just in case other folks look into this. The extra field above is only part of the values.yaml file in you helm chart. That shouldn’t be added to the daemonset template yaml file.