I encountered a failure while attempting to render an image using an HTML graphics panel and utilizing MQTT as the data source after upgrading to version 10.2.2.
version before : v.9.2 upgrade to version 10.2.2
version image render : 3.6.4
1 Like
I have seen the same issue which I tracked down to first appearing in 10.2.0:
Initial config:
Grafana 10.1.5
grafana-image-renderer @ 3.9.0
vertamedia-clickhouse-datasource @ 2.5.4
Log output on successful image rendering:
Dec 6 11:14:18 test grafana[18238]: logger=rendering renderer=plugin t=2023-12-06T11:14:18.467932854Z level=info msg=Rendering path="d-solo/wom-ext-5minagg-rssac-report/rssac-reporting?orgId=1&from=1701216000000&to=1701302400000&var-Server=IMRS&panelId=1&width=1000&height=600&tz=UTC&theme=light"
Dec 6 11:14:20 test grafana[18238]: logger=live t=2023-12-06T11:14:20.660721417Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/wom-ext-5minagg-rssac-report address=grafana/dashboard/uid/wom-ext-5minagg-rssac-report
After upgrading to Grafana 10.2.0. Image rendering then fails with error handler=notfound followed by 404 and eventual time out. Log output:
Dec 6 11:22:04 test grafana[1745]: logger=rendering renderer=plugin t=2023-12-06T11:22:04.493135044Z level=info msg=Rendering path="d-solo/wom-ext-5minagg-rssac-report/rssac-reporting?orgId=1&from=1701216000000&to=1701302400000&var-Server=IMRS&panelId=1&width=1000&height=600&tz=UTC&theme=light"
Dec 6 11:22:06 test grafana[1745]: logger=context userId=0 orgId=1 uname= t=2023-12-06T11:22:06.312343443Z level=info msg="Request Completed" method=GET path=/stats/public/plugins/vertamedia-clickhouse-datasource/module.js status=404 remote_addr=[::1] time_ms=11 duration=11.885544ms size=38867 referer="http://localhost:3000/d-solo/wom-ext-5minagg-rssac-report/rssac-reporting?from=1701216000000&height=600&orgId=1&panelId=1&render=1&theme=light&to=1701302400000&tz=UTC&var-Server=IMRS&width=1000" handler=notfound
Dec 6 11:22:06 test grafana[1745]: logger=plugin.grafana-image-renderer t=2023-12-06T11:22:06.322564005Z level=error msg="Browser console error" msg="Failed to load resource: the server responded with a status of 404 (Not Found)" url="http://localhost:3000/stats/public/plugins/vertamedia-clickhouse-datasource/module.js?_cache=2.5.4"
Dec 6 11:22:06 test grafana[1745]: logger=live t=2023-12-06T11:22:06.342323153Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/wom-ext-5minagg-rssac-report address=grafana/dashboard/uid/wom-ext-5minagg-rssac-report
...
Dec 6 11:23:04 test grafana[1745]: logger=plugin.grafana-image-renderer t=2023-12-06T11:23:04.830431143Z level=error msg="Error while trying to prepare page for screenshot" err="TimeoutError: Navigation timeout of 60000 ms exceeded\n at Timeout.<anonymous> (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/util/Deferred.js:27:33)\n at listOnTimeout (node:internal/timers:564:17)\n at process.processTimers (node:internal/timers:507:7)" url="http://localhost:3000/d-solo/wom-ext-5minagg-rssac-report/rssac-reporting?orgId=1&from=1701216000000&to=1701302400000&var-Server=IMRS&panelId=1&width=1000&height=600&tz=UTC&theme=light&render=1"
- 10.2.1 and 10.2.2 give the same error as 10.2.0
- Downgrade to 10.1.5 and image rendering is successful again
I have this issue on 10.2.2 with specifically the briangann-gauge-panel. Other panels render correctly.
logger=context userId=0 orgId=1 uname= t=2023-12-18T11:37:16.254392342Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=[::1] time_ms=2 duration=2.332682ms size=0 referer= handler=/api/live/ws
logger=context userId=0 orgId=1 uname= t=2023-12-18T11:37:16.578653497Z level=info msg=“Request Completed” method=GET path=/grafana/public/plugins/briangann-gauge-panel/module.js status=404 remote_addr=[::1] time_ms=7 duration=7.418371ms size=42147 referer=“http://localhost:3000/d-solo/26a-V5Wiz/<–removed–>?from=now-24h&height=140&openVizPicker=&orgId=1&panelId=17&refresh=30s&render=1&theme=light&to=now&tz=Europe%2FLondon&width=140” handler=notfound
logger=plugin.grafana-image-renderer t=2023-12-18T11:37:16.582067761Z level=error msg=“Browser console error” msg=“Failed to load resource: the server responded with a status of 404 (Not Found)” url=“http://localhost:3000/grafana/public/plugins/briangann-gauge-panel/module.js?_cache=2.0.1”
After some debugging into the GET requests made during image rendering it became clear this was related to the use of a sub path to server. By altering our configuration we were able to get image rendering to work again
Original config
We set our Grafana servers up several years ago to serve fro a sub path called stats and we also use an nginx reverse proxy. Our original configuration for nginx used something like the following:
location /stats {
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:3000/;
rewrite /stats/(.*) /$1 break;
…
And in our grafana.ini file we had:
root_url = https://<domain>/stats
;serve_from_sub_path = false
Issue seen
On 10.1.5 the call to GET the plugin (which worked) was:
/public/plugin/vertmedia-clickhouse-datasource/module.js
But on 10.2.0 this changed to the following which failed:
/stats/public/plugin/vertmedia-clickhouse-datasource/module.js
Updated working config
After updating our configuration in the following way we are still serving from a subpath but now the image rendering works.
- Modify the
nginxconfiguration to match the suggested mechanism described on this page that does not require a rewrite Run Grafana behind a reverse proxy | Grafana Labs i.e. in our case
upstream stats {
server localhost:3000;
}
server {
location /stats/ {
proxy_set_header Host $http_host;
proxy_pass http://stats;
…
}
- And then update our
grafana.inifile to have the following:
root_url = https://<domain>:3000/stats
serve_from_sub_path = true
Bug?
The documentation page states
If your Grafana configuration does not set
serve_from_sub_pathto true then you need to add a rewrite rule to each location block:
however making these changes to my updated (working) configuration broke image rendering again. So it seem there is issue with 10.2.0 + later versions and this configuration.
1 Like
My setup is similar, except I’m using apache as the reverse proxy. That server_from_sub_path configuration has caused me issues in the past - another grafana update a couple of years ago. I originally has it set as true, then image rendering broke and I set it to false which fixed the issue (despite the fact I am serving from a sub path - https://myhostname.me/grafana). It never made sense to me at the time, but as it fixed the issue I never bothered following up further.
As I mentioned in my earlier post, I only have issues with one panel type though…
Thanks all for creating this post and providing very detailed insights about it (especially @sdsin 
)
Though I am not an expert with the Image rendering component but I do know it causes a lot of configuration issues (breaks a lot with a new version).
As it seems that you have narrowed it down then I would propose and appreciate it if you could open a bug report about this in our official GitHub repository using this link.
Provide all the information (you can simply copy/paste your data + screenshot from here to there).
Lastly, paste the link to your GitHub issue here in this post so that all other community users can also track it. I will also add more additional information to it
@usman.ahmad Thanks for the feedback - I will go ahead and create a bug report but I also wanted to add to the discussion here that with my working configuration, an upgrade to 10.2.3 breaks the image rendering in a different way 
When trying to generate an image using the Direct link rendered image link on the Share dialog I now get an immediate 404 error in the browser. This is because in 10.2.2 that link contained the subpath e.g.
https://<domain>/<sub_path>/render/d-solo/...
whereas on 10.2.3 it does not:
https://<domain>/render/d-solo/...
Note that obtaining the image with a curl command or browser request with the correct path still works on 10.2.3, it is just the target of that link that is broken.
With upgrade to 10.4.2, the issue I had with the briangann-gauge-panel and using a subpath and apache reverse proxy has been fixed. Thanks!