Grafana-Agent - sharding in static mode (not using operator)

Grafana Alloy
1

Has anyone gotten this to work? I just can’t seem to get an env var to work in a statefulset setup. It appears to work if I either export or inline set an env var to the shard value.. like 0, or 1, or 2.. but fails when I try to interpolate from hostname or POD_NAME (if using downward API). I am missing some secret sauce here ..
Have tried all sorts of variants like:

      - command:
          - /bin/sh
          - -c
          - |
            set -e
            export POD_ORDINAL=${POD_NAME##*-}
            exec /usr/bin/grafana-agent \
              -config.file=/etc/agent/agent.yaml \
              -config.expand-env=true \
              -enable-features=integrations-next \
              -server.http.address=0.0.0.0:80
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
2

Brief update - I tested further by exporting some variables and using them to relabel known fields and it worked. So I believe the export and expanding work.. I am either using the hashmod relabels incorrectly or it doesn’t work as expected in “non-operator” setups.

3

Final update - while using relabel configs to output the values of my expanded env variables and the value of the hashmod action the sharding started to work. After messign around for FAR too long I discovered that if you use the target_label of the hashmod in another relabel it will suddenly work as intended. I’m at a loss as to why this would be (maybe some sorting or prioritizing bug?) …

          relabel_configs:
            # Sharding
            - action: hashmod
              modulus: ${SHARDS}
              source_labels: [__address__]
              target_label: __tmp_hash
            - action: keep
              regex: ${POD_ORDINAL}
              source_labels:
                - __tmp_hash

              # This is bonkers, but it works
            - action: replace
              source_labels: [__tmp_hash]
              target_label: shard_hash_ignore

If I remove the replace action config my grafana pods will all scrape the same endpoints and mimir will complain about duplicate metrics. If I keep it, it appears to work as intended, only scraping where the hashmod modulus == pod ordinal.

bizarre to say the least.