Environment:
Grafana installed using the kube-prometheus-stack helm chart: v47.1.0
Basic authentication is disabled on Grafana
Azure AD authentication is enabled on Grafana
Kubernetes environment: v1.26.3(Azure Kubernetes service)
I have defined values for both the default data source as well as a value in the additional data sources in the kube-prometheus-stack helm chart . No matter what i do, i could not get the default datasource loaded automatically using the data source side car. Even killing the grafana pod did not really help
The grafana datasource side car always returned the following error:
POST request sent to http://localhost:3000/api/admin/provisioning/datasources/reload. Response: 401 Unauthorized
I stumbled upon the issue on Github: [grafana] Data sources are not loaded properly when Grafana is configured with anonymous log in · Issue #981 · grafana/helm-charts · GitHub, where it was suggested that Grafana needs to be enabled with basic authentication for the above POST request to work.
I then enabled basic authentication for Grafana. However, i still continued to face the same Unauthorized error.
I then made the following changes in the kube-prometheus-stack helm chart:
skipReload: true
initDatasources: true
With the above changes, i now have an additional init continaer which loads the datasource at the time of creation of the grafana pod. With the above changes, i am able to get the data source updated(However, inspite of the above changes, i still have to sometimes kill the grafana pod as updation of the datasource only causes the associated config map to be updated and not the grafana pod to be reloaded).
So my question really is, how do we configure Grafana to reload changed data sources automatically using the kube-promehtues-stack helm chart configuration?
I am also attaching a template run of my kube-prometheus-stack helm chart for reference(I have intentionally deleted the non essential parts to keep the file as short as possible)
Any suggestions would be very much appreciated.
Regards,
Kiran Hegde
---
# Source: kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: grafana-6.57.4
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "9.5.5"
app.kubernetes.io/managed-by: Helm
name: kube-prometheus-stack-grafana
namespace: observability
---
# Source: kube-prometheus-stack/charts/kube-state-metrics/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: kube-state-metrics-5.8.1
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: kube-state-metrics
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "2.9.2"
release: kube-prometheus-stack
name: kube-prometheus-stack-kube-state-metrics
namespace: observability
imagePullSecrets:
- name: pmtsecret
---
# Source: kube-prometheus-stack/charts/prometheus-node-exporter/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-prometheus-stack-prometheus-node-exporter
namespace: observability
labels:
helm.sh/chart: prometheus-node-exporter-4.18.1
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "1.6.0"
jobLabel: node-exporter
release: kube-prometheus-stack
imagePullSecrets:
- name: pmtsecret
---
# Source: kube-prometheus-stack/templates/alertmanager/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-prometheus-stack-alertmanager
namespace: observability
labels:
app: kube-prometheus-stack-alertmanager
app.kubernetes.io/name: kube-prometheus-stack-alertmanager
app.kubernetes.io/component: alertmanager
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
automountServiceAccountToken: true
imagePullSecrets:
- name: pmtsecret
---
# Source: kube-prometheus-stack/templates/prometheus-operator/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-prometheus-stack-operator
namespace: observability
labels:
app: kube-prometheus-stack-operator
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
app.kubernetes.io/component: prometheus-operator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
imagePullSecrets:
- name: pmtsecret
---
# Source: kube-prometheus-stack/templates/prometheus/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-prometheus-stack-prometheus
namespace: observability
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/name: kube-prometheus-stack-prometheus
app.kubernetes.io/component: prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
imagePullSecrets:
- name: pmtsecret
---
# Source: kube-prometheus-stack/templates/thanos-ruler/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-prometheus-stack-thanos-ruler
namespace: observability
labels:
app: kube-prometheus-stack-thanos-ruler
app.kubernetes.io/name: kube-prometheus-stack-thanos-ruler
app.kubernetes.io/component: thanos-ruler
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
imagePullSecrets:
- name: pmtsecret
# Source: kube-prometheus-stack/templates/alertmanager/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: alertmanager-kube-prometheus-stack-alertmanager
namespace: observability
labels:
app: kube-prometheus-stack-alertmanager
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
data:
alertmanager.yaml: "blah"
---
# Source: kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: grafana-6.57.4
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "9.5.5"
app.kubernetes.io/managed-by: Helm
name: kube-prometheus-stack-grafana-config-dashboards
namespace: observability
data:
provider.yaml: |-
apiVersion: 1
providers:
- name: 'sidecarProvider'
orgId: 1
folder: ''
type: file
disableDeletion: false
allowUiUpdates: false
updateIntervalSeconds: 30
options:
foldersFromFilesStructure: false
path: /tmp/dashboards
---
# Source: kube-prometheus-stack/charts/grafana/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-prometheus-stack-grafana
namespace: observability
labels:
helm.sh/chart: grafana-6.57.4
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "9.5.5"
app.kubernetes.io/managed-by: Helm
data:
grafana.ini: |
[analytics]
check_for_updates = true
[auth.anonymous]
enabled = true
hide_version = true
org_name = Main Org.
org_role = Viewer
[auth.azuread]
allow_assign_grafana_admin = true
allow_sign_up = true
auth_url = https://blah
auto_login = false
client_id = blah
client_secret = blah
enabled = true
name = Azure AD
role_attribute_strict = false
scopes = openid email profile
skip_org_role_sync = false
token_url = https://blah
use_pkce = true
[grafana_net]
url = https://grafana.net
[log]
level = debug
mode = console
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[security]
admin_password = blah
[server]
domain = ''
root_url = https://blah.com/grafana
serve_from_sub_path = true
[users]
viewers_can_edit = true
---
# Source: kube-prometheus-stack/templates/grafana/configmaps-datasources.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-prometheus-stack-grafana-datasource
namespace: observability
labels:
grafana_datasource: "1"
app: kube-prometheus-stack-grafana
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: kube-prometheus-stack
app.kubernetes.io/version: "47.1.0"
app.kubernetes.io/part-of: kube-prometheus-stack
chart: kube-prometheus-stack-47.1.0
release: "kube-prometheus-stack"
heritage: "Helm"
data:
datasource.yaml: |-
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
url: http://thanos-query:9090/thanos-query
access: proxy
isDefault: true
jsonData:
httpMethod: POST
timeInterval: 30s
- name: Alertmanager
type: alertmanager
uid: alertmanager
url: http://kube-prometheus-stack-alertmanager.observability:9093/alertmanager
access: proxy
jsonData:
handleGrafanaManagedAlerts: false
implementation: prometheus
- access: proxy
editable: false
isDefault: false
name: Loki
type: loki
url: http://grafana-loki-gateway
version: 1
---
