Grafana and prometheus issue

Hello Team,

All of a sudden alerts started coming with below error note.

tsdb.HandleRequest() error Post "http://localhost:9090/api/v1/query_range": dial tcp [127.0.0.1:9090](http://127.0.0.1:9090/): connect: connection refused

We checked that prometheus is running properly on 9090 port and is able to fetch the metrics from other servers as well. But this alert is coming intermittently and sometimes clears also.

We also checked grafana logs and found this snippet but doesn’t seem relevant to this particular error

t=2020-11-23T08:48:30+0800 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://localhost:3000/d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15&render=1" error="Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=129 error="Rendering failed: Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"

PFA image for the alert

Environment:

  • Grafana version: v7.1.5 (9893b8c53d)
  • Data source type & version: prometheus2-2.20.1-2.el7
  • OS Grafana is installed on: cpe:2.3:o:amazon:amazon_linux:2
  • User OS & Browser: Windows 10 & Chrome
  • Grafana plugins: grafana-image-renderer @ 2.0.0, grafana-piechart-panel @ 1.6.0, snuids-trafficlights-panel @ 1.4.5

Image render plugin is already installed on the server. Please help to correct this.

Regards
Nitin G

I even tried to update the both the versions to below but no fate. Please help.

Grafana to v7.3.3
Prometheus2 to v2.22.2-1.el7

Never Mind … It is resolved by replacing Data Source Prometheus URL in Grafana UI from http://localhost:9090 to http://:9090