Hello Team,
All of a sudden alerts started coming with below error note.
tsdb.HandleRequest() error Post "http://localhost:9090/api/v1/query_range": dial tcp [127.0.0.1:9090](http://127.0.0.1:9090/): connect: connection refused
We checked that prometheus is running properly on 9090 port and is able to fetch the metrics from other servers as well. But this alert is coming intermittently and sometimes clears also.
We also checked grafana logs and found this snippet but doesn’t seem relevant to this particular error
t=2020-11-23T08:48:30+0800 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://grafana.staged-by-discourse.com/d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15&render=1" error="Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=129 error="Rendering failed: Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"
PFA image for the alert
Environment:
- Grafana version: v7.1.5 (9893b8c53d)
- Data source type & version: prometheus2-2.20.1-2.el7
- OS Grafana is installed on: cpe:2.3:o:amazon:amazon_linux:2
- User OS & Browser: Windows 10 & Chrome
- Grafana plugins: grafana-image-renderer @ 2.0.0, grafana-piechart-panel @ 1.6.0, snuids-trafficlights-panel @ 1.4.5
Image render plugin is already installed on the server. Please help to correct this.
Regards
Nitin G