401 error on the alerts sidecar

Hi,

I’m getting 401 authentication error on /api/admin/provisioning/alerting/reload
Double checked: the username / password is correct, basic authentication is enabled

  • What Grafana version and what operating system are you using?
    Grafana 9.3.6 deployed by Helm in K8s

  • What are you trying to achieve?
    Automatic alerts provisioning

  • How are you trying to achieve it?
    Config map with alert yaml

  • What happened?
    All good except the reload api call, throwing 401

  • What did you expect to happen?
    Load alerts in Grafana

  • Can you copy/paste the configuration(s) that you are having problems with?

[analytics]
check_for_updates = false
[auth.basic]
enabled = true
[auth.ldap]
allow_sign_up = false
config_file = /etc/grafana/ldap.toml
enabled = false
[auth.proxy]
auto_sign_up = true
enabled = true
header_name = X-Auth-Request-Email
header_property = email
headers = Email:X-Auth-Request-Email
sync_ttl = 60
whitelist =
[database]
host = postgresql
name = grafana
password = $__env{pgPass}
ssl_mode = disable
type = postgres
user = $__env{pgUser}
[dataproxy]
keep_alive_seconds = 60
logging = true
timeout = 600
[feature_toggles]
tempoBackendSearch = true
tempoSearch = true
tempoServiceGraph = true
[grafana_net]
url = https://grafana.net
[log]
mode = console
[paths]
data = /var/lib/grafana/data
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[server]
domain = ''
[unified_alerting]
enabled = true
ha_peers = grafana-headless:9094
[users]
auto_assign_org = true
auto_assign_org_role = Admin
viewers_can_edit = true
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Defaulted container "grafana-sc-alerts" out of: grafana-sc-alerts, grafana-sc-dashboard, grafana-sc-datasources, grafana-sc-notifiers, grafana, init-chown-data (init)
{"time": "2023-03-01T15:34:22.900808+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2023-03-01T15:34:22.900936+00:00", "level": "WARNING", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2023-03-01T15:34:22.901013+00:00", "level": "DEBUG", "msg": "Selected resource type: ('secret', 'configmap')"}
{"time": "2023-03-01T15:34:22.901095+00:00", "level": "INFO", "msg": "Loading incluster config ..."}
{"time": "2023-03-01T15:34:22.901529+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://10.96.0.1:443' loaded..."}
{"time": "2023-03-01T15:34:22.901604+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2023-03-01T15:34:22.901662+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
{"time": "2023-03-01T15:34:22.901726+00:00", "level": "DEBUG", "msg": "Ignore already processed resource version will not be enabled."}
{"time": "2023-03-01T15:34:27.914649+00:00", "level": "DEBUG", "msg": "Working on ADDED configmap cloudprober/alerts"}
{"time": "2023-03-01T15:34:27.915213+00:00", "level": "DEBUG", "msg": "Found 'data' on configmap"}
{"time": "2023-03-01T15:34:27.915281+00:00", "level": "DEBUG", "msg": "Writing alerts.yaml"}
{"time": "2023-03-01T15:34:27.915411+00:00", "level": "DEBUG", "msg": "Contents of alerts.yaml haven't changed. Not overwriting existing file"}
{"time": "2023-03-01T15:35:32.919603+00:00", "level": "DEBUG", "msg": "Working on ADDED configmap cloudprober/alerts"}
{"time": "2023-03-01T15:35:32.919890+00:00", "level": "DEBUG", "msg": "Found 'data' on configmap"}
{"time": "2023-03-01T15:35:32.919950+00:00", "level": "DEBUG", "msg": "Writing alerts.yaml"}
{"time": "2023-03-01T15:35:32.920055+00:00", "level": "DEBUG", "msg": "Contents of alerts.yaml haven't changed. Not overwriting existing file"}
{"time": "2023-03-01T15:36:37.925474+00:00", "level": "DEBUG", "msg": "Working on ADDED configmap cloudprober/alerts"}
{"time": "2023-03-01T15:36:37.925766+00:00", "level": "DEBUG", "msg": "Found 'data' on configmap"}
"time": "2023-03-01T15:49:08.332577+00:00", "level": "DEBUG", "msg": "Working on MODIFIED configmap cloudprober/alerts"}
{"time": "2023-03-01T15:49:08.332881+00:00", "level": "DEBUG", "msg": "Found 'data' on configmap"}
{"time": "2023-03-01T15:49:08.332939+00:00", "level": "DEBUG", "msg": "Writing alerts.yaml"}
{"time": "2023-03-01T15:49:08.332939+00:00", "level": "DEBUG", "msg": "Writing alerts.yaml"}
{"time": "2023-03-01T15:49:08.337772+00:00", "level": "DEBUG", "msg": "POST request sent to http://localhost:3000/api/admin/provisioning/alerting/reload. Response: 401 Unauthorized {\"message\":\"invalid username or password\",\"traceID\":\"\"}\n"}

reinitialization of the postgres db fixed the issues