Error messages since instalation of 9.4.1

installed (updated) grafana from 9.3.6 to v9.4.1 (fd67ab151d) → success

wget https://dl.grafana.com/oss/release/grafana_9.4.1_armhf.deb
sudo dpkg -i grafana_9.4.1_armhf.deb

following error messages appeared after a while (repeats every 15 min), I haven’t got any grafana alerts defined:

logger=alertmanager org=1 t=2023-03-01T09:21:17.76653514+01:00 level=error msg="Running maintenance failed" err="context canceled"
logger=alertmanager org=1 t=2023-03-01T09:36:17.766832842+01:00 level=error msg="Running maintenance failed" err="context canceled"

noticed (webmin monitoring) change in running processes:

  • which grafana-server says:
/usr/sbin/grafana-server
  • systemd sevice file says:
ExecStart=/usr/share/grafana/bin/grafana server                                     \
                            --config=${CONF_FILE}                                   \
                            --pidfile=${PID_FILE_DIR}/grafana-server.pid            \
                            --packaging=deb                                         \
                            cfg:default.paths.logs=${LOG_DIR}                       \
                            cfg:default.paths.data=${DATA_DIR}                      \
                            cfg:default.paths.plugins=${PLUGINS_DIR}                \
                            cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}
  • webmin running processes says:
  29704	grafana	43 minutes ago	/usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfil ...
      29819	grafana	42 minutes ago	/var/lib/grafana/plugins/frser-sqlite-datasource/gpx_sqlite-datasource_linux_arm
      29828	grafana	42 minutes ago	/var/lib/grafana/plugins/redis-datasource/redis-datasource_linux_arm
1 Like

hi @prussell :wave:

could you share the part concerning unified alerting in your config file ?

it might also help to know the following details:

  • OS Grafana is installed on:

  • User OS

thank you

Raspberry pi 3B with Raspberry Pi OS Lite Release date: February 21st 2023, fresh install of everything, grafana 9.3.6 on 26/02/2023 (4 days ago), copied over the database (sqlite3) from previous system, upgrade, as indicated, today 01/03/2023.
uname -a: Linux raspberrypi 5.15.84-v7+ #1613 SMP Thu Jan 5 11:59:48 GMT 2023 armv7l GNU/Linux

I’ve made no changes in this section:

#################################### Unified Alerting ####################
[unified_alerting]
#Enable the Unified Alerting sub-system and interface. When enabled we'll migrat
e all of your alert rules and notification channels to the new system. New alert
 rules will be created and your notification channels will be converted into an
Alertmanager configuration. Previous data is preserved to enable backwards compa
tibility but new data is removed.```
;enabled = true

# Comma-separated list of organization IDs for which to disable unified alerting
. Only supported if unified alerting is enabled.
;disabled_orgs =

...skipping 1 line
# Specify the frequency of polling for admin config changes.
# The interval string is a possibly signed sequence of decimal numbers, followed
 by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;admin_config_poll_interval = 60s

# Specify the frequency of polling for Alertmanager config changes.
# The interval string is a possibly signed sequence of decimal numbers, followed
 by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;alertmanager_config_poll_interval = 60s

# Listen address/hostname and port to receive unified alerting messages for othe
r Grafana instances. The port is used for both TCP and UDP. It is assumed other
Grafana instances are also running on the same port. The default value is `0.0.0
.0:9094`.
;ha_listen_address = "0.0.0.0:9094"

# Listen address/hostname and port to receive unified alerting messages for othe
r Grafana instances. The port is used for both TCP and UDP. It is assumed other
Grafana instances are also running on the same port. The default value is `0.0.0
.0:9094`.
;ha_advertise_address = ""

# Comma-separated list of initial instances (in a format of host:port) that will

...skipping 1 line
;ha_peers = ""

# Time to wait for an instance to send a notification via the Alertmanager. In HA, each Grafana instance will
# be assigned a position (e.g. 0, 1). We then multiply this position with the timeout to indicate how long should
# each instance wait before sending the notification to take into account replication lag.
# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;ha_peer_timeout = "15s"

# The interval between sending gossip messages. By lowering this value (more frequent) gossip messages are propagated
# across cluster more quickly at the expense of increased bandwidth usage.
# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;ha_gossip_interval = "200ms"

# The interval between gossip full state syncs. Setting this interval lower (more frequent) will increase convergence speeds
# across larger clusters at the expense of increased bandwidth usage.
# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;ha_push_pull_interval = "60s"

# Enable or disable alerting rule execution. The alerting UI remains visible. This option has a legacy version in the `[alerting]` section that takes precedence.
;execute_alerts = true

# Alert evaluation timeout when fetching data from the datasource. This option has a legacy version in the `[alerting]` section that takes precedence.
# The timeout string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;evaluation_timeout = 30s

# Number of times we'll attempt to evaluate an alert rule before giving up on that evaluation. This option has a legacy version in the `[alerting]` section that takes precedence.
;max_attempts = 3

# Minimum interval to enforce between rule evaluations. Rules will be adjusted if they are less than this value  or if they are not multiple of the scheduler interval (10s). Higher values c
an help with resource management as we'll schedule fewer evaluations over time. This option has a legacy version in the `[alerting]` section that takes precedence.
# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
;min_interval = 10s

[unified_alerting.reserved_labels]
# Comma-separated list of reserved labels added by the Grafana Alerting engine that should be disabled.
# For example: `disabled_labels=grafana_folder`
;disabled_labels =

inspired by the reaction here (mellieA: I’m curious if you create an alert if this will clear?), I searched, found and followed these instructions, the error message no longer appears.

sudo sed -i -r '/^\[alerting\]/,/;enabled = false/ s|;enabled = false|enabled = false|' /etc/grafana/grafana.ini
sudo sed -i -r '/^\[unified_alerting\]/,/;enabled = true/ s|;enabled = true|enabled = false|' /etc/grafana/grafana.ini
1 Like

:wave:

for those users that have the same issue

level=error msg="Running maintenance failed" err="context canceled"

upgrading to 9.4.3 should fix the issue
reference here