Image-renderer preventing grafana-server 8.2.0 to start

  • What Grafana version and what operating system are you using?
    8.2.0 on Debian 11
  • What are you trying to achieve?
    Start Grafana server
  • How are you trying to achieve it?
    Service Start
  • What happened?
    Updated to the last version. Grafana entered a start loop.
  • What did you expect to happen?
    Grafana working
  • Can you copy/paste the configuration(s) that you are having problems with?
    No config change
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Can´t reach the GUI
  • Did you follow any online instructions? If so, what is the URL?
    No

Attaching log:

t=2021-10-07T15:39:47-0300 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2021-10-07T15:39:47-0300 lvl=info msg="App mode production" logger=settings
t=2021-10-07T15:39:47-0300 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-10-07T15:39:47-0300 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-10-07T15:39:47-0300 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=841.006µs
t=2021-10-07T15:39:47-0300 lvl=info msg="Starting plugin search" logger=plugins
t=2021-10-07T15:39:47-0300 lvl=info msg="Registering plugin" logger=plugins id=input
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=natel-plotly-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=petrslavotinek-carpetplot-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Starting ClickHouse datasource backend..." logger=plugins.backend pluginId=vertamedia-clickhouse-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=vertamedia-clickhouse-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg=Profiler logger=plugins.backend pluginId=grafana-influxdb-flux-datasource enabled=false
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-influxdb-flux-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-simple-json-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=natel-influx-admin-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=farski-blendstat-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=michaeldmoore-annunciator-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-clock-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-image-renderer
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-piechart-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=jdbranham-diagram-panel
t=2021-10-07T15:39:52-0300 lvl=warn msg="Some plugin scanning errors were found" logger=plugins errors="plugin 'jasonlashua-prtg-datasource' is unsigned, plugin 'btplc-trend-box-panel' is unsigned"
t=2021-10-07T15:39:52-0300 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
t=2021-10-07T15:39:52-0300 lvl=info msg="Writing PID file" logger=server path=/run/grafana/grafana-server.pid pid=25792
t=2021-10-07T15:39:52-0300 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
t=2021-10-07T15:39:52-0300 lvl=eror msg="Stopped background service *rendering.RenderingService" logger=server reason="Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2021-10-07T15:39:52-0300 lvl=warn msg="plugin failed to exit gracefully" logger=plugins.backend pluginId=grafana-image-renderer
t=2021-10-07T15:39:52-0300 lvl=eror msg="Server shutdown" logger=server error="*rendering.RenderingService run error: Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2021-10-07T15:39:53-0300 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
t=2021-10-07T15:39:53-0300 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
t=2021-10-07T15:39:53-0300 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-10-07T15:39:53-0300 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2021-10-07T15:39:53-0300 lvl=info msg="App mode production" logger=settings
t=2021-10-07T15:39:53-0300 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-10-07T15:39:53-0300 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-10-07T15:39:53-0300 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=2.115013ms
t=2021-10-07T15:39:53-0300 lvl=info msg="Starting plugin search" logger=plugins
t=2021-10-07T15:39:53-0300 lvl=info msg="Registering plugin" logger=plugins id=input

Same problem with ubuntu 18.04.6 LTS
Service don’t start after update to 8.2.0

Yep,

Same problem using RockyLinux 8.4, the work around is to uninstall grafana-image-renderer plugins, but I don’t know the effect, since my dashboard working fine…

I tried to uninstall and grafana server up and running, then when install the grafana-image-renderer plugins, grafana stopped, even the plugins goes to version 3.2.1
The alert with image will broken as alert to Telegram.

Regards,
Fadjar Tandabawana

i can confirm this issue too. Im on ubuntu server 21.04 with grafana 8.2.0. ive tried to update the image renderer plugin with no change. uninstalling the plugin appears to have resolved this issue

Hi All,

think we should make an issue for this in the repo :+1:

could y’all provide more detailed reproduction steps? Installing a fresh 8.2.0 standalone binary, I can install image-renderer using the new plugin platform inside the UI.

Upgrading/Updating using a package manager? Updating an instance with image-renderer` already installed, I presume… From what version Grafana?

No doubt everyone is hitting a bug: just want to collect some clearer steps for reproduction

Installing with grafana-cli results in grafana not being able to start, with the log I posted before.

Installing via GUI yields this error, while trying to render a direct share image:

image

Everything began when I upgraded to 8.2.0. If I uninstall the plugin via grafana-cli, grafana starts with no problems. I upgraded grafana as usual, with apt update/upgrade on my Debian installation. I don’t remember the previous version, but It was 8 with certain, and not too old. If I can recover wich version I had, please tell me how.

Thanks

so there is an issue right now with 8.2.0 and systemd. I’m seeing it on my end, when I try to update grafana on ubuntu 18.04 to 8.2.0 and the service hangs.

Can you try this workaround posted for that other issue and see what happens?:

2 Likes

Thank you. The workaround worked for me.

1 Like

this should get patched Monday. in the meantime here is a note in the changelog on the breaking change:

Thank you so much. This one works for me after updating to the latest Grafana (08 Oct 2021 update).
OS: Ubuntu 20.04 LTS
Grafana: 8.2
Image Renderer: 3.2.1