Image-renderer preventing grafana-server 8.2.0 to start

dmayan · October 7, 2021, 6:41pm

What Grafana version and what operating system are you using?
8.2.0 on Debian 11
What are you trying to achieve?
Start Grafana server
How are you trying to achieve it?
Service Start
What happened?
Updated to the last version. Grafana entered a start loop.
What did you expect to happen?
Grafana working
Can you copy/paste the configuration(s) that you are having problems with?
No config change
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Can´t reach the GUI
Did you follow any online instructions? If so, what is the URL?
No

Attaching log:

t=2021-10-07T15:39:47-0300 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-10-07T15:39:47-0300 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2021-10-07T15:39:47-0300 lvl=info msg="App mode production" logger=settings
t=2021-10-07T15:39:47-0300 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-10-07T15:39:47-0300 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-10-07T15:39:47-0300 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=841.006µs
t=2021-10-07T15:39:47-0300 lvl=info msg="Starting plugin search" logger=plugins
t=2021-10-07T15:39:47-0300 lvl=info msg="Registering plugin" logger=plugins id=input
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=natel-plotly-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=petrslavotinek-carpetplot-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Starting ClickHouse datasource backend..." logger=plugins.backend pluginId=vertamedia-clickhouse-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=vertamedia-clickhouse-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg=Profiler logger=plugins.backend pluginId=grafana-influxdb-flux-datasource enabled=false
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-influxdb-flux-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-simple-json-datasource
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=natel-influx-admin-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=farski-blendstat-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=michaeldmoore-annunciator-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-clock-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-image-renderer
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=grafana-piechart-panel
t=2021-10-07T15:39:52-0300 lvl=info msg="Registering plugin" logger=plugins id=jdbranham-diagram-panel
t=2021-10-07T15:39:52-0300 lvl=warn msg="Some plugin scanning errors were found" logger=plugins errors="plugin 'jasonlashua-prtg-datasource' is unsigned, plugin 'btplc-trend-box-panel' is unsigned"
t=2021-10-07T15:39:52-0300 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
t=2021-10-07T15:39:52-0300 lvl=info msg="Writing PID file" logger=server path=/run/grafana/grafana-server.pid pid=25792
t=2021-10-07T15:39:52-0300 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
t=2021-10-07T15:39:52-0300 lvl=eror msg="Stopped background service *rendering.RenderingService" logger=server reason="Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2021-10-07T15:39:52-0300 lvl=warn msg="plugin failed to exit gracefully" logger=plugins.backend pluginId=grafana-image-renderer
t=2021-10-07T15:39:52-0300 lvl=eror msg="Server shutdown" logger=server error="*rendering.RenderingService run error: Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
t=2021-10-07T15:39:53-0300 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
t=2021-10-07T15:39:53-0300 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
t=2021-10-07T15:39:53-0300 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-10-07T15:39:53-0300 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2021-10-07T15:39:53-0300 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-10-07T15:39:53-0300 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2021-10-07T15:39:53-0300 lvl=info msg="App mode production" logger=settings
t=2021-10-07T15:39:53-0300 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-10-07T15:39:53-0300 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-10-07T15:39:53-0300 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=2.115013ms
t=2021-10-07T15:39:53-0300 lvl=info msg="Starting plugin search" logger=plugins
t=2021-10-07T15:39:53-0300 lvl=info msg="Registering plugin" logger=plugins id=input

stevenes · October 8, 2021, 1:57am

Same problem with ubuntu 18.04.6 LTS
Service don’t start after update to 8.2.0

fadjar340 · October 8, 2021, 4:48am

Yep,

Same problem using RockyLinux 8.4, the work around is to uninstall grafana-image-renderer plugins, but I don’t know the effect, since my dashboard working fine…

I tried to uninstall and grafana server up and running, then when install the grafana-image-renderer plugins, grafana stopped, even the plugins goes to version 3.2.1
The alert with image will broken as alert to Telegram.

Regards,
Fadjar Tandabawana

sniff122 · October 8, 2021, 2:09pm

i can confirm this issue too. Im on ubuntu server 21.04 with grafana 8.2.0. ive tried to update the image renderer plugin with no change. uninstalling the plugin appears to have resolved this issue

mattabrams · October 8, 2021, 7:26pm

Hi All,

think we should make an issue for this in the repo

mattabrams · October 8, 2021, 8:23pm

could y’all provide more detailed reproduction steps? Installing a fresh 8.2.0 standalone binary, I can install image-renderer using the new plugin platform inside the UI.

Upgrading/Updating using a package manager? Updating an instance with image-renderer` already installed, I presume… From what version Grafana?

No doubt everyone is hitting a bug: just want to collect some clearer steps for reproduction

dmayan · October 8, 2021, 9:33pm

Installing with grafana-cli results in grafana not being able to start, with the log I posted before.

Installing via GUI yields this error, while trying to render a direct share image:

Everything began when I upgraded to 8.2.0. If I uninstall the plugin via grafana-cli, grafana starts with no problems. I upgraded grafana as usual, with apt update/upgrade on my Debian installation. I don’t remember the previous version, but It was 8 with certain, and not too old. If I can recover wich version I had, please tell me how.

Thanks

mattabrams · October 8, 2021, 9:43pm

so there is an issue right now with 8.2.0 and systemd. I’m seeing it on my end, when I try to update grafana on ubuntu 18.04 to 8.2.0 and the service hangs.

Can you try this workaround posted for that other issue and see what happens?:

github.com/grafana/grafana

Systemd: Upgrading from 8.0.6 to 8.2.0 broke grafana

opened 04:57PM - 07 Oct 21 UTC

vsenapathy

priority/critical area/backend

> **Note from Grafana team:** We're on this and will release a patch release tha…t addresses this. In the meantime, you can [add an override to remove the syscall filter from the grafana-server service](https://github.com/grafana/grafana/issues/40162#issuecomment-938060240). ---  **What happened**: Upgraded my dev Grafana from 8.0.6 to 8.2.0 by running the following commands 1. Took a backup of the DB 2. sudo apt-get update 3. sudo apt-get upgrade 4. grafana-cli plugins update-all 5. sudo service grafana-server restart 6. systemctl status grafana-server.service ● grafana-server.service - Grafana instance Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled) Active: failed (Result: signal) since Thu 2021-10-07 16:51:50 UTC; 7s ago Docs: http://docs.grafana.org Process: 25507 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=deb cfg:default.paths.logs=${LOG_DIR} c Main PID: 25507 (code=killed, signal=SYS) Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5. Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: Stopped Grafana instance. Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: grafana-server.service: Start request repeated too quickly. Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 07 16:51:50 ip-10-216-5-98 systemd[1]: Failed to start Grafana instance. **What you expected to happen**: On reboot, expected grafana to come up properly without issues **How to reproduce it (as minimally and precisely as possible)**: Upgrade from 8.0.6 to 8.2.0 **Anything else we need to know?**: **Environment**: - Grafana version: 8.2.0 - Data source type & version: Influx - OS Grafana is installed on: Ubuntu 18.04 - User OS & Browser: Mac and Google chrome - Grafana plugins: - Others:

dmayan · October 8, 2021, 10:22pm

Thank you. The workaround worked for me.

mattabrams · October 8, 2021, 10:50pm

this should get patched Monday. in the meantime here is a note in the changelog on the breaking change:

michael630angelo · October 9, 2021, 1:38am

Thank you so much. This one works for me after updating to the latest Grafana (08 Oct 2021 update).
OS: Ubuntu 20.04 LTS
Grafana: 8.2
Image Renderer: 3.2.1

system · October 7, 2022, 6:41pm

This topic was automatically closed after 365 days. New replies are no longer allowed.