How to configure a prometheus datasource using a datasource.yaml

Grafana Configuration
1
  • What Grafana version and what operating system are you using?

v11.1.0 on Kubernetes using the Helm chart.

  • What are you trying to achieve?

I am trying to configure a datasource using the datasource sidecar. I have an existing datasource correctly configured via the UI which I want to put into code.

  • How are you trying to achieve it?

I have a secret which I am configuring as per the documentation.

Rather than writing it from scratch, I also thought to try find any existing configuration written to the container but there is nothing in the filepaths listed in other documentation such as in /etc/grafana/provisioning/datasources.

I also hoped to be able to find some yaml/json from a button on the UI like you can with dashboards but I could also not figure out if this is possible.

  • What happened?

I don’t know what to write in the configuration. I have looked everywhere for reference docs on the format of the YAML. None of it is complete for what I need to configure such as the settings for AWS SIGv4 authentication using the default SDK creds

2
1 Like
3

Thanks for this tip, I got the structure of this data source from the API though I am not able to see it in Grafana. According to the logs of the datasource sidecar, the file is being written to /etc/grafana/provisioning/datasources but it does not show up once Grafana starts. However, a dashboard using the dashboard sidecar is successfully loaded this way.

I have tried both a minimal set of properties (omitting what I presume are the defaults), as well as putting the JSON output from the API directly into the file/secret. The logs of the Grafana container do not contain anything regarding this except for the following log:

logger=context userId=0 orgId=0 uname= t=2024-07-05T11:15:38.286802148Z level=info msg="Request Completed" method=POST path=/api/admin/provisioning/datasources/reload status=401 remote_addr=[::1] time_ms=1 duration=1.178236ms size=121 referer= handler=/api/admin/provisioning/datasources/reload status_source=server
4

Please provide reproducible example what are you doing.

5

Here’s a minimal example of what I am trying to achieve.

  • Deploy the Secret with the datasource as shown in datasouce.yaml
  • Deploy the Grafana Helm chart with the values below
  • Load the Grafana UI and expect the data source to be configured

values.yaml (for Grafana Helm chart)

appVersion: 11.1.0
sidecar:
  datasources:
    enabled: true
env:
  AWS_SDK_LOAD_CONFIG: true
  GF_AUTH_SIGV4_AUTH_ENABLED: true

datasource.yaml

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-datasource
  labels:
    grafana_datasource: ""
type: Opaque
stringData:
  my-prometheus.yaml: |-
    apiVersion: 1
    datasources:
      - name: my-prometheus
        type: prometheus
        access: proxy
        url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxx1234
        isDefault: true
        basicAuth: false
        jsonData:
          sigV4Auth: true
          sigV4AuthType: default
          sigV4Region: eu-west-1
        editable: false
6

Pls check sidecar pod logs (you may increase also logLevel for sidecar). Also sigv4 env variables must be configured for grafana pod, not for sidecar.

7

It looks like the issue was due to a persistent volume and the grafana database on it which was holding onto data not congruous with the bootstrapped data. I’ve opted to turn persistence off when using this sidecar bootstrapping approach. What is the recommendation on this?

8

That’s opinionated question. My opininion: I hate PVC and files in this case. I provision SaaS DB (e. g. Amazon Autora) and I use that in the Grafana. I use included SaaS tools: daily snapshots/restore, master/slave replication/switch.
Of course K8S only engineers will be preferring pure K8S solution only.

Lovely example of PVC issues: Upgrade grafana to 11.0.0

1 Like