Hi,
we have recently updated our Grafana deployment to version 7. As the internal image rendering code has vanished there, we have added the dockerized image renderer to out setup as described here: https://github.com/helm/charts/issues/21959#issuecomment-640653320. This means, grafana and the renderer work directly on the same isolated network without any caching being involved. Since then, sometimes, rendering images for slack notifications fails.
What we see is that Grafana reports the following error messages:
- Failed to send request to remote rendering service.
- context deadline exceeded
- Failed to get render key from cache
The renderer reports errors along the following lines:
- Failed to load resource: the server responded with a status of 401 (Unauthorized)
- TypeError: Cannot read property ‘statusCode’ of undefined
This only happens sometimes and other slack notifications work correctly.
Any ideas what could be the cause of this?