Loki secret take control on secret I've provided

[lekspek]

After installing Hashicorp vault on our cluster with vault secrets operator, it’s time to add keep S3 credentials on secret.
I’ve added a secret into the cluster, but moment after helm install the secret get a new owner and it’s looks like this now:

ownerReferences:
  - apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultStaticSecret
    name: vault-kv-app
    uid: d39043fa-560b-40b7-8ce2-2c353f00232a
  - apiVersion: monitoring.grafana.com/v1alpha1
    blockOwnerDeletion: true
    kind: GrafanaAgent
    name: loki
    uid: eda13595-bb61-496d-94d1-883bb772068c

From the vault side I get error that this owner isn’t authorized and now it can’t be changed from the vault, which make it static.
few questions.

1. Why GrafanaAgent take control?
2. How I can avoid this?
3. Any good practice to use Hashicorp vault with Loki?

[mkamsika]

Hi lekspek,

Did you manage to resolve this. I’m using flux to deploy grafana-loki and using sealed secrets.
The issue I am having is similar.
I create a sealed secret called loki-secrets containing my s3 credentials
I follow Storing s3 accessKeyID and secretAccessKey securely to map the secrets values as environment variables
Sealed secrets creates the secret loki-secrets but the secrets data is removed and the grafana-agent-operator takes over the secret

apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: "2024-03-01T09:30:22Z"
  labels:
    app.kubernetes.io/managed-by: grafana-agent-operator
  name: loki-secrets
  namespace: logging
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: loki-secrets
    uid: 416ec23a-d50b-4470-aed6-fa5870f6a087
  - apiVersion: monitoring.grafana.com/v1alpha1
    blockOwnerDeletion: true
    kind: GrafanaAgent
    name: loki
    uid: ce95503e-7350-4717-905a-0907a918f413
  resourceVersion: "479732"
  uid: ceeea6ab-ec5d-4d53-81e8-01abd2c36e5e
type: Opaque

The pods then fails. Sealed secrets cannot recreate the missing data.